Transcript Digital Preservation Standards for e
Dr. Dinesh Katre Chief Investigator Centre of Excellence for Digital Preservation Associate Director & HOD Human-Centred Design & Computing Group C-DAC, Pune, INDIA.
Background Information
Indo-US Workshop on International Trends in Digital Preservation March 23-24, 2009
Panel recommendations for India’s National Digital Preservation Programme Proceedings of Indo US Workshop
Background Information National Meet of Expert Group Members held at C-DAC, Pune
Dr. A. K. Chakravarti, Chairman of Expert Group Dr. Dinesh Katre, Principal Investigator (C-DAC) Dr. Gautam Bose, National Informatics Centre Ms. Renu Budhiraja, e-Governance Division, DIT Mr. Sumnesh Joshi, Unique Identification Authority of India Dr. Meena Gautam, National Archives of India Ms. Debjani Nag, Controller of Certifying Authorities Mr. Vakul Sharma, Supreme Court Dr. Mukul Sinha, Expert Software Consultants Dr. Kamalini Dutt, Doordarshan Dr. Ramesh C. Gaur, Indira Gandhi National Centre for the Arts Ms. Manju Mathur, All India Radio Dr. S. B. Bhattacharyya, e-Health Consultant Dr. Usha Munshi, Indian Institute of Public Administration Dr. Vandana Sinha, American Institute of Indian Studies Dr. A. Moorthy, Defence Scientific Information & Documentation Centre, DRDO Mr. Ramchandra Budihal, WIPRO Mr. Sanjeev Kumar Gupta, IBM Mr. V.V.S. Nageswara Rao, National Remote Sensing Centre, Dept of Space Mr. Zia Saquib, Centre for Development of Advanced Computing Mr. Ashok Kapoor, Reserve Bank of India Mr. Patrick Kishor, State Bank of India Mr. Sukhdev Singh, National Informatics Centre Mr. Vivek K. Srivastava, e-Governance Division, DIT Ms. Seema Sridhar, Life Insurance Corporation of India Mr. N. S. Mani, National Archives of India Mr. V. H. Jadhav, National Film Archives of India Dr. V. C. V. Rao, Centre for Development of Advanced Computing Dr. Y.K. Somayajulu, National Institute of Oceanography
May 20-21, 2010
National Study Report on
Digital Preservation Requirements of India
Volume – I Recommendations for National Digital Preservation Programme
Chapter 1. Need of Digital Preservation Chapter 2. Scope of Digital Preservation in India Chapter 3. Recommendations for NDPP
Volume – II Position Papers by National Expert Group Members
23 Status Reports Stakeholder Recommendations Short Term (3Yrs) and Long Term Actions (10Yrs)
Legal Imperatives
Information Technology Act 2000
….. conditions for retention of electronic records are - accessibility so as to be usable for a subsequent reference; retention in the format in which it was originally generated, to represent accurately the information originally generated, with the details, which will facilitate the identification of the origin, destination, date and time of dispatch or receipt of such electronic record.
Public Records Act 1993
….. every record creating agency of the central government, any ministry, department or office of the Government must provide proper arrangement, maintenance and preservation of public records.
Right to Information Act 2005
… every public authority is obliged to maintain all its records duly catalogued and indexed .... ensure that all records that are appropriate to be computerized are, within a reasonable time, computerized and connected through a network all over the country on different systems so that access to such records is facilitated.
The Indian Evidence Act (Medico Legal Requirements)
The medical records need to be preserved up to certain length of time as they are treated as documentary evidence..... responsibility of hospital to keep medical records is up to duration of 5years for outpatient department and inpatient medical record (case sheet MLC and Non MLC) is up to 10 years. Where there is chance of litigation arising for medical purpose of negligence, record should be preserved for at least 25 years .... Privacy, photo copying and unauthorized access to medical record without patient's consent and destruction or alteration of a medical record by an unauthorized person are sensitive matters
Banking Regulation Act, 1949 and Bankers’ Books Evidence Act, 1891
A banking company is responsible for preservation its books, accounts and other documents .. - the safeguards adopted to prevent and detect unauthorized change of data; retrieval of data, storage, integrity and accuracy of data...
Central Secretariat Manual of e-Procedure
Two copies of the archived digital records has to be maintained one at the location of Disaster Recovery Site if any and the other with the Data Center or “Digital Record room” to be set up as part of e-Office ......... requirements of inter-connection between relevant e-files, use of standard taxonomy, metadata, digital signature for establishing authenticity of e-file, security, preservation, retrieval and long term e record management.
Preservation & Archival
Legal Imperatives
Information Technology Act 2000
….. conditions for retention of electronic records are - accessibility so as to be usable for a subsequent reference; retention in the format in which it was originally generated, to represent accurately the information originally generated, with the details, which will facilitate the identification of the origin, destination, date and time of dispatch or receipt of such electronic record.
Public Records Act 1993
….. every record creating agency of the central government, any ministry, department or office of the Government must provide proper arrangement, maintenance and preservation of public records.
Right to Information Act 2005
… every public authority is obliged to maintain all its records duly catalogued and indexed .... ensure that all records that are appropriate to be computerized are, within a reasonable time, computerized and connected through a network all over the country on different systems so that access to such records is facilitated.
The Indian Evidence Act (Medico Legal Requirements)
The medical records need to be preserved up to certain length of time as they are treated as documentary evidence..... responsibility of hospital to keep medical records is up to duration of 5years for outpatient department and inpatient medical record (case sheet MLC and Non MLC) is up to 10 years. Where there is chance of litigation arising for medical purpose of negligence, record should be preserved for at least 25 years .... Privacy, photo copying and unauthorized access to medical record without patient's consent and destruction or alteration of a medical record by an unauthorized person are sensitive matters
Banking Regulation Act, 1949 and Bankers’ Books Evidence Act, 1891
A banking company is responsible for preservation its books, accounts and other documents .. - the safeguards adopted to prevent and detect unauthorized change of data; retrieval of data, storage, integrity and accuracy of data...
Central Secretariat Manual of e-Procedure
Two copies of the archived digital records has to be maintained one at the location of Disaster Recovery Site if any and the other with the Data Center or “Digital Record room” to be set up as part of e-Office ......... requirements of inter-connection between relevant e-files, use of standard taxonomy, metadata, digital signature for establishing authenticity of e-file, security, preservation, retrieval and long term e record management.
Digital Preservation (What?)
Cultural Heritage data E-Records Government data Video data Audio data Scientific Data Insurance & Banking data Websites Educational data Copyright © CoE-DP / HCDC Group C-DAC Pune Our Digital Universe Large volume of e-Records are being generated through the 27 mission mode projects of National e-Governance Action Plan (NeGP) across the country The digital organizations information from generated by domains such as government, cultural heritage, science and education, audio, video, films, health, insurance, banking, etc are part of our digital universe or each one is a digital universe in its own right As per the “Digital Universe Study Report” by International Data Corporation (IDC), 2010 -
Includes the digital information from government, enterprise and social segments
Estimated Size of Digital Universe in 2010 1.2 million Petabytes or 1.2 Zettabytes Estimated Size of Digital Universe in 2020 35 Zettabytes
(As all major forms of media radio, print would have –voice, TV, completed the journey from analog to digital.)
Estimated Size of Unprotected needing protection in 2020 Data 18,000 Exabytes (appox. 18 Zettabytes)
Digital Preservation (Why?)
Need
to Preserve the Digital Universe
Copyright © CoE-DP / HCDC Group C-DAC Pune The fear of Digital Dark Age by 2020 Digital information of today is digital heritage for tomorrow’s generations Technological Imperatives Non-tangible bitstream form of digital data Obsolescence of file formats Obsolescence of computer software hardware Obsolescence of storage devices and media Failure of devices Physical threats & Legal Imperatives Digital Preservation of Records Act (Need has been identified) Electronic soon) Service Delivery Bill (Expected Information Technology Act Right To Information Act Public Records Act Indian Evidence Act Banking Regulation Act Banking Books Evidence Act Copyright Act / Digital Rights Central Secretariat Manual for e-Procedure / e office Absence of digital preservation strategies Lack of awareness about digital obsolescence No digital preservation policy No standards, guidelines and best practices
Digital Preservation (How?)
National Initiatives by Department of Information Technology National Digital Preservation Infrastructure Centre of Excellence for Digital Preservation at C-DAC Pune National Report on Digital Preservation Requirements of India Recommendations for National Digital Preservation Programme by the subject experts and stakeholders Digital Preservation Strategies Trustworthy Digital Repositories Audit and Certification Digital Preservation Course Curriculum Digital Practices Preservation Standards and Best Digital Preservation Tools and Technologies Digital Preservation Policy Awareness Goal
Preserving Our Digital Universe for Access
Copyright © CoE-DP / HCDC Group C-DAC Pune Nature of Activity Domain specific solutions Dynamic, cyclic, continuously evolving Long term commitment Dependable and trustworthy Interoperable Protective
Goal
Preserving Our Digital Universe for Access
Copyright © CoE-DP / HCDC Group C-DAC Pune Vision Statement for National Digital Preservation Programme Pursue the goal of long-term digital preservation of digital information (i.e.
inclusive of born digital and reformatted digital content) across diverse domains; by creating the tools, technologies, best practices and sustainable infrastructure; to ensure usability of digital information through passage of time and obsolescence.
Objectives of Centre of Excellence (C-DAC Pune) Project Duration: April 2011 to March 2014 A) Conduct research and development in digital preservation to produce the required tools, technologies, guidelines and best practices.
B) Develop the pilot digital preservation repositories and provide help in nurturing the network of Trustworthy Digital Repositories (National Digital Preservation Infrastructure) as a long-term goal.
C) Define the digital preservation standards by involving the experts from stakeholder organizations, consolidate and disseminate the digital preservation best practices generated through various projects under National Digital Preservation Programme, being the nodal point for pan-India digital preservation initiatives.
D) Provide inputs to Department of Information Technology in the formation of national digital preservation policy and strategy by identifying and selecting the activities for the National Digital Preservation Programme.
E) Spread awareness about the potential threats and risks due to digital obsolescence and the digital preservation best practices.
Scope of CoE-DP Project
Portal for National Digital Preservation Program (NDPP) Digital Preservation Best Practices and Standards for e-Governance T O O L S & O G I E S T E C H N O L Digital Repository Portals for Access to Designated Users Digital Repository 01 Digital Preservation of Government Archives Digital Repository 02 Archival Science Library Science Digital Preservation of Born Digital Records Digital Repository 03 Digital Content Preparing for Valid Submission Information Package Domain Specific Digital Preservation and Archival Systems Creating a complete Archival Information Package Offering appropriate Delivery Information Package to Designated Users Digital Preservation Research & Development Authenticity Management and Digital Preservation Audit
NeGP: e-District Mission Mode Project Offer Electronic Delivery of district and sub-district level services Digitization of legacy Data Establish 100,000 Citizen Service Centres 15 Pilot e-Districts are implemented India approx. 600 districts 31 State Data Centres are being established
DP Case Study 01 Digital Preservation of Born Digital Records
Services offered through e-District MMP
DP Case Study 01 Digital Preservation Born Digital Records
Services planned to be delivered through e-district MMP include:
Creation and distribution of certificates for income, domicile, caste, Birth, Death etc.
Arms Licenses, Driving Licenses, etc.
Public Distribution System (PDS): Issue of Ration Card, etc.
Social Welfare Schemes: Disbursement of old-age pensions, family pensions, widow pensions, etc.
Marriage Registration, Land Records, etc.
Assessment of taxes: Property tax, and other government taxes.
RTI: Online filing and receipt of information relating to the Right to Information Act
Other MMPs:
E-Courts E-Office implementation throughout government offices.
State Data Centres (SDCs) / Potential Digital Repositories
7 8 9 10 11 12 13 4 5 6 1
Sl.No
2 3 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
State/UT
Andaman Andhra Pradesh Arunachal Pradesh Assam Bihar Chattisgarh Goa Gujarat Haryana Himachal Pradesh J&K Jharkhand Karnataka Kerala Lakshadweep Maharashtra Madhya Pradesh Manipur Meghalaya Mizoram Nagaland Orissa Puducherry Punjab Rajasthan Sikkim Tamil Nadu Tripura Uttar Pradesh Uttarakhand West Bengal
DP Case Study 01 State Implementing Agency (Nodal Government Agency)
SOVTECH (Society for Vocational Technical Education) Andhra Pradesh Technology Services Limited (APTSL), Hyderabad Arunachal Pradesh State Council for Science and Technology Assam Electronics Development Corporation Ltd (AMTRON), Assam Bihar State Electronics Development Corporations Ltd (BSEDC) Chattisgarh Infotech Promotion Society (ChiPS), Chattisgarh InfoTech Corporation of Goa Ltd, Goa Gujarat Informatics Limited, Gujarat Haryana State Electronics Development Corporation Ltd (HARTRON) Society for IT& e-governance, HP Jammu and Kashmir e-Governance Agency (JaKeGA) Jharkhand Agency for Promotional Information (JAPIT), Jharkhand Centre for e-Governance (CEG)
Digital Preservation Born Digital Records
Kerala State Information Technology Mission, Kerala Lakshadweep Information Technology Services Ltd (LITSS) SETU Maharashtra (State Level Apex SETU Society of Government of Maharashtra) Madhya Pradesh State Electronics Development Corporations (MPSEDC), Madhya Pradesh Manipur Science & Technology Council, Manipur Meghalaya Information Technology Society (MITS) Mizoram State e-Governance Society Nagaland e governance society (NeGS) Orissa Computer Application Centre (OCAC), Orissa Puducherry e-governance society (PsGS), Puducherry Punjab State e-governance Society, Punjab Rajasthan State Agency for Computer Services (RajCOMP), Rajasthan Centre for Research and Training in Informatics, (CRTI), Sikkim Electronics Corporation of Tamil Nadu Ltd (ELCOT), Tamil Nadu Tripura State Computerization Agency, Tripura Uttar Pradesh Development Systems Corporation Ltd, Uttar Pradesh Technology Development Agency, Department of Information Technology, Govt of Uttarakhand West Bengal Industry Development Corporation Ltd (WBEIDC)
Single database file comprising of millions of e-records
• • • • • •
Digital Preservation Diagnostics
Unwieldy to manage due to constantly inflating size Difficult to migrate the data Vendor dependence in case of proprietary database DB obsolescence If one DB file is corrupted all records could be lost Unacceptable as single e-record
Existing e-District Model
E-Records are getting produced without consideration for preservation Collects data from citizen
Collects data from citizen
Enhanced e-District Model
Testing the Applicability of Dublin Core Elements for e-Certificates
Sr.
Nos.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Label Contributor Coverage Creator Date Description Format Identifier Language Publisher Relation Rights Source Subject Title Type Definition
An entity responsible for making contributions to the resource.
The spatial or temporal topic of the resource, the spatial applicability of the resource, or the jurisdiction under which the resource is relevant.
An entity resource.
primarily responsible for A point or period of time associated with an event in the lifecycle of the resource.
An account of the resource.
making the The file format, physical medium, or dimensions of the resource.
An unambiguous reference to the resource within a given context.
A language of the resource.
An entity available.
responsible A related resource.
for making the resource Information about rights held in and over the resource.
A related resource from which the described resource is derived.
The topic of the resource.
A name given to the resource.
The nature or genre of the resource.
Common Elements of Cataloging Metadata for e-Certificates (draft)
10 11 12 7 8 9 4 5 6
Sr. Nos.
1 2
E-Cert - PDI - Block 01 of 6 - Standard cataloging elements for e-records (mandatory) Label Record Identifier Type of Record Definition
It is the reference number or record number.
The classification of record in terms of whether it is a certificate or passport or document or license or card or land record or office memorandum, tender, office order, receipt / bill, etc.
EC Inputs
3
Date of Record
The official date on which the record got completed.
Issuing Authority State Place
The office / title of authority which authorizes the record.
The name of Indian state where the record is produced.
The city/town/village where the office which produced the record is located.
Pin Code Issued To
The number associated with the place.
The name of person / organization which is the intended user of the record.
ID with No.
Associated Person organization /
A unique number associated with the person such as Employee No., UID No., PAN number, Election ID No., Passport No. Registration No. License No. or other ID numbers acceptable to the record creating agency, etc.
Language of Record Validity Period Retention Duration
The languages of information contained in the record.
Duration for which the record is considered valid based on some rule or law.
The duration for which the record must be preserved.
Collaboration with National Archives of India
DP Case Study 02 Digital Preservation of Government Archives
Digitization at National Archives of India
Type of collections Official files Volumes Maps and cartographic items Volume of data
38,75,332 64,221 1,10,332 3,601
Bills assented to by the President(s) of India Treaties Rare manuscripts
1,065 2,442
Microfilms Microfiches Total digitized data Cataloguing metadata entered so far
35,000 (4.9 million pages approximately) 9000 [0.9 million pages]
5, 00,000 files in TIFF and PDF formats 22,00,000 records
Relational Model of Government Archives
DP Case Study 02 Digital Preservation of Government Archives
Collaboration with Indira Gandhi National Centre for Arts
DP Case Study 03 Digital Preservation of Cultural Digital Data Type of collections
Cultural Digital Data with IGNCA
Volume of data Rare manuscripts from different manuscripts repositories in India are digitized Photographs and slides collected from the institutions like ACSAA, British Library, Victoria and Albert Museum Paintings
2,00,000 with many folios (JPG, TIFF and
PDF)
1,00,000 JPG, PhotoCD, TIFF images 3000 in JPG, TIFF formats
Rare books Audio / Video documentation of 10,000 hours available in obsolete analogue media such as U-matic, VHS tapes, Betacam, etc
1700 (approx. 2 Lakh pages in JPG, TIFF and PDF) Several hours of audio video digitized in WAV / MP3 and MPEG I / II formats currently stored in DVDs
Long Term Goal (Difference between TDR & Data Centre)
Physical setup Location, building architecture Server hardware / racks / KVM Storage hardware (SAN, NAS) Backup device Network Infrastructure (Switches, routers, UTM) Raised flooring Connectivity and bandwidth Disaster recovery site Physical Security Biometric Security Network Security (Firewall) Motion detection for lighting CCTV digital recorders Humidity control Cooling Fire suppression Smoke detection Redundant power supply
Thank You