The regulator’s focus on the critical objectives of data retention: Risk down, cost down, data utility up

Data is not the new oil, contrary to the popular assertion. Unlike oil, data is not finite, exclusive, depletable or limited to specific use. “Good data” can be synthesised, repurposed and reused. Its favourable attributes make it a major asset of great value which, however, comes with increased regulatory attention, storage and governance costs and legal risk.

As Australia’s Privacy Commissioner announces the updated guidance about the retention, destruction and de-identification of personal information for not-for-profits,[1] even resource-constrained organisations are reminded that “Indefinite retention of all personal information is unlikely to satisfy an entity’s APP obligations”. Moreover, it can result in exacerbated “risk or impact of data breaches”[2] and adverse regulatory action.

Data governance including a data retention policy is key. With data sharing for innovation being promoted at international and governmental levels, good data governance will help organisations unlock the value of data and explore opportunities across the data-driven economy.

Apart from significantly reducing the cost of storage and data management and limiting exposure to legal risk, there are environmental benefits too – less data helps reduce emissions and landfill (as server hard drives are often physically destroyed upon deletion for data security reasons).

Getting data retention right will eventually have a direct positive impact on the organisation’s balance sheet.

Data governance is key to forming a data retention policy

Data governance is a set of rules about all activities around the data lifecycle including the collection, storage, use and disposal of enterprise data. Data retention is a key aspect of it, and its concerns cut across the organisation, touching upon operational needs, information security, legal compliance and data privacy.

Data governance is often pushed aside by spending priorities, such as revenue generating activities. Meanwhile, simply keeping all data is easier than having to manage it, even if it means seeing the cost of data redundancy rise and ignoring the intrinsic value of data.

Understanding the value of enterprise data is important. Apart from its critical use for various business purposes, emerging analytics and AI data marketplaces may offer compensation for datasets suitable for analytics and the development of AI.

From a simplified point of view, some necessary data governance steps in designing a data retention policy will relate to:

• Data mapping and data inventory across all assets
• Identifying data-reliant functions and activities across the organisation
• Considering data availability, storage and interoperability issues
• Onboarding third party data governance, management and erasure systems
• Automating long-term data retention and data erasure across all assets

If data is collected for targeted lawful purposes, properly classified, we know where to find it, it is reasonably accurate to achieve our purpose, reasonably utilised, repurposed and reformatted for secondary purposes in a timely manner, tracked throughout its lifecycle and erased across all assets and devices when no longer needed, the organisation is succeeding in formulating and executing a data retention policy. This may translate into the recognition of new capital assets, identifying new savings and financial gains.

Setting a data retention period is hard

Data retention strategies are essential in maximising value and reducing the risk of data. They strive on the “less is more” principle. The benefits of good data retention lie in purging useless, unnecessary or otherwise “bad data”, and typically there is plenty of it around.

Often organisations view the risk of not having data as greater than the risk of having data. However, the law does not reward data hoarding. Often storage of data gives rise to risk such as data breaches, unauthorised access to data in breach of policies, interference with the privacy of staff or customers, increased storage costs and the natural staff turnaround results in loss of organisational familiarity about stored files further diminishing their value.

Caution often drives organisations to set long retention periods inspired by the prescribed statutory limitation periods, e.g. 7 years from the cause of action arising, during which one can bring a legal claim. However, the age of a document may have little bearing on its evidential value. Besides, a document can be as helpful as it can be damaging in proving the organisation’s case. Adopting such a blanket rule for all documents is difficult to reconcile with any logic or legal requirement. Nevertheless, it is a popular starting point; albeit not one to be recommended

Data retention strategies must often find a common ground between varying corporate interests, such as:

• Audit interest in keeping certain records for a minimum period (sometimes prescribed by law, e.g. in relation to tax records) to enable statutory, regulatory or internal auditing.
• Cyber security standards such as ISO27001 which mandates business continuity policies such as duplicating data backups, data-driven network security policies such as activity logging and monitoring and access controls such as recognising user patterns for authentication purposes.
• Data governance and compliance interest in preserving data for regular use and applying consistent rules to all data assets.
• Finance team’s interest in reducing the cost of electronic and physical data storage.
• Legal interest in preserving evidence for contemplated litigation and destroying data the deliberate or inadvertent disclosure of which could give rise to liability.
• Operational needs to retain as much data as possible for ever changing immediate and ‘what if?’ future organisational goals, e.g. enhancing consumer profiles, gaining market insights, identifying procedural deficiencies, developing services, monetising data, etc.

In practice, setting retention periods is a complex task. Expect plenty of internal battles before being able to formulate a data retention policy. Starting with achievable short-term goals and longer time periods which are gradually tightened and granularised might work best, as long as this is based on sound justifications.

… and there is a bit of law too

The following Australian Privacy Principles (APP) under the Privacy Act 1988 (Cth) are relevant to data retention of personal information:

• APP 1.2 requires that organisations take reasonable steps to implement compliance practices, procedures and systems.
• APP 10.2 requires that organisations take reasonable steps to ensure accuracy, completeness and relevance of personal information.
• APP 11.2 requires that organisations take reasonable steps to destroy or de‑identify personal information no longer needed.

Various other laws prescribe rules about data retention. The Freedom of Information Act 1982 (Cth) prescribes certain publication of government data by federal agencies and ministers; the Corporations Act 2001 (Cth) requires organisations to keep accounting records, registers and meeting minutes; Fair Work Act 2009 (Cth) requires organisations to keep certain employee records; the Workplace Surveillance Act 2005 (NSW) requires organisations to take reasonable safeguards including data erasure to protect surveillance records; the Telecommunications (Interception and Access) Act 1979 establishes a data retention scheme for telecommunications service providers, etc.

Data erasure is a technological process

Being one of the hardest decisions in data governance for which no one wishes to take responsibility, data erasure presents a number of challenges; the obvious one being whether deleting data in the near future will present a cost saving and risk mitigation, or instead result in a loss of opportunity or inability to defend against claims.

From a practical point of view, data indexing across all domains may be difficult considering data storage architecture such as data lakes, warehouses and siloed data marts, virtual or physical storage facilities, on-prem or remote, centralised or decentralised, long-term or instant access storage, etc. However, without such indexing, data erasure cannot be implemented.

This is exacerbated by the unreliability of conventional erasure techniques. A “DIY” approach might not result in the desired risk mitigation. Third party enterprise data erasure software and drive eraser tools conforming to standards such as NIST 800-88, IEEE 2883-2022, DoD 5220.22-M/ECE and possibly certified under the Australian Information Security Evaluation Program (AISEP),[3] integrated with major data governance and data storage providers, will be required to wipe all accessible data storage blank. Getting erasure wrong might mean that the data removal will not result in the expected risk mitigation.

Conclusion

Data retention is hard. However, without data governance and a data retention policy, significant insights, monetisation, risk mitigation, cost-saving opportunities and environmental benefits could be lost.

Besides, organisations have a legal obligation to engage with data governance, and failing to do so could result in an infringement of cyber, data privacy and confidentiality obligations and give rise to reputational, regulatory and litigation risk.

KHQ can help organisations find practical ways of meeting legal requirements in data retention, exploring data monetisation, facilitating technology onboarding and increasing cyber risk preparedness. Our subscription service is designed to meet our clients’ long-term support needs.

This article was written by Alex Dittel (Principal Solicitor).

Want Data Privacy, Cyber & Digital updates delivered straight to your inbox? Click here to subscribe. 

[1] OAIC’s guidance “Privacy for not-for-profits, including charities” updated on 22 October 2024 (link).

[2] LinkedIn post, Carly Kind, 22 October 2024 (link).

[3] Australian Information Security Evaluation Program, Australian Signals Directorate (link).