TS-53: Privacy and Data Protection

This technical standard covers general principles and practices for protecting the privacy of individuals and the data that is collected about them.

Privacy by design

Applications MUST be designed with privacy and data protection embedded as core design principles. This is known as privacy by design or privacy by default. This concept emerged in the 1970s and it means, simply, "data protection through technology design". It means that data processing procedures and data protection controls are considered at the design stage of an IT system, or at the design stage of any subsequent change made to that system, rather than these concerns being treated as separate features that are added on later.

The principle of privacy by design has since been added to data protection legislation such as the European Union’s General Data Protection Regulation (GDPR). GDPR explicitly requires that technical and organizational measures for protecting data be proactively designed into business processes and systems, and that businesses anticipate and put in place prevention mechanisms for privacy breaches before they happen.

The principles of privacy by design can be summarized by the following seven statements:^[1]

1. Be proactive about privacy, not reactive. Anticipate and prevent privacy breaches before they happen.
2. Build maximum privacy in to default settings. Users may explicitly opt-in to reduce privacy settings, but maximum privacy safeguards must be the default. Thus, if users do nothing, they are protected.
3. Embed privacy into the design of IT systems and business processes. Privacy should be integral to a system, not an add-on.
4. Privacy may need to be balanced against other legitimate interests and objectives, but the goal is to achieve a positive-sum (not zero-sum) outcome, in which privacy is not traded off against other objectives.
5. End-to-end security: embed strong encryption and other security measures to the complete lifecycle of data.
6. Be open and transparent about your privacy policies and standards. Inform users about how their data is processed and stored. Allow for independent verification of your privacy practices.
7. Give users control over their data, and make it easy for them to manage their privacy settings.

Personally identifiable information (PII)

Personally identifiable information (PII) is any information that can be used to identify an individual. It includes, but is not limited to:

• Names.
• Postal addresses.
• Email addresses.
• Phone numbers.
• Passport numbers, social security numbers, and other government-issued identifiers.
• Bank account numbers and other financial credentials.

Combinations of values can also form personally identifiable information. For example, neither a date of birth nor a postcode can, on their own, identify an individual. However, when combined, they can.

Some values are considered to be "borderline" PII, such as IP addresses. Whether borderline values should be handled as PII depends on the context. For example, an IP address may be considered PII if it is logged with other information such as timestamps and user agent strings.

PII MUST NOT be stored anywhere other than production databases and their backup systems.

For non-production systems, synthetic (dummy) data MUST be used in place of real PII. Alternatively, a hash of real PII MAY be used, as long as a strong cipher is used (making the hash irreversible without the cipher) and the cipher is handled as a secret – see TS-52: Security and Secrets Management.

Where it is not easy to distinguish between real and synthetic PII, the data SHOULD be clearly annotated as being synthetic.

Automated tools SHOULD be used to detect and prevent the storage of PII in non-production systems such as source code repositories. For example, pre-commit hooks can be enabled in Git to run secret scanning tools.

Data retention

Applications that store data ("data applications") MUST have a data retention policy, which MUST be documented as part of the application’s design.

The data retention policy MUST define a data retention schedule for each discrete type of data stored by the application. The data retention policy MUST clearly define the retention period for each type of data, and the conditions under which data will be automatically deleted. In addition, the data retention policy MUST document the data types that may be stored indefinitely, which MUST include technical data such as logs and monitoring data.

Data applications MUST have an automated mechanism for deleting data as per the data retention policy. A common implementation pattern is to use scheduled tasks. Data deletion schedules and operations MUST be planned to minimize impact on the performance or availability of production systems.

In addition, there MUST be a documented manual process for finding and deleting data that is outside of its retention period. The manual process MUST be undertaken at regular intervals to verify that the automatic deletion mechanism is functioning as expected.

Automated and manual data deletion processes MAY need to involve caches, replicas, backups and failovers, as well as primary data sources, depending on the data synchronization strategies in use.

For auditing purposes, applications MUST log data deletion events. Such logs will capture information such as:

• When the process ran, and when it completed.
• Who initiated the run (user or automated agent).
• Which data was deleted. (This may be a simple matter of recording the number of entities deleted for each data type.)
• Any errors that were encountered. (The errors SHOULD also be captured in a separate error log.)

Data deletion audit logs MUST be easily accessible and searchable for reporting purposes.

Data redaction

Applications that store personally identifiable information (PII) MUST provide a mechanism for users to request the redaction or erasure of their data. This requirement is mandated by data protection legislation in many jurisdictions, most notably the European Union’s General Data Protection Regulation (GDPR), which enshrines in law the right to erasure (also known as the "right to be forgotten") in Article 17.

GDPR and similar regulations require that organizations erase personal data when a user makes such a request, unless there is a legal basis for retaining it. Organizations must provide a mechanism for users to exercise this right, and must respond to such requests in a timely manner (typically within 30 days under GDPR).

Scope of redaction

When a user requests the redaction of their data, applications MUST identify all personal data belonging to that user and remove it from all systems in which it is stored. This includes:

• Primary databases and data stores.
• Backup systems and snapshots.
• Caches.
• Replicas and failovers.
• Log files and audit trails (where PII has been captured).

Data associated with the user that is retained for legitimate legal or business purposes (such as transaction records required for accounting, tax, or fraud prevention) MAY be retained. OPTIONALLY, this held records may be anonymized or pseudonymized, to remove association with the individual.

Implementation requirements

Applications MUST have an automated mechanism to handle data redaction requests. A common implementation pattern is to mark user accounts as deleted or redacted, then run a scheduled process that removes all associated data from production systems and backups.

Data redaction processes MUST be planned and tested to ensure they do not impact system performance or availability. In particular, care must be taken when redacting data from backup systems, as restoration of a backup may inadvertently restore redacted data.

For auditing and compliance purposes, applications MUST log all data redaction events. Such logs MUST capture:

• When the redaction request was received.
• Who initiated the request (the user or an authorized representative).
• What data was redacted or deleted.
• When the redaction process completed.
• Any errors or exceptions encountered.

Data redaction audit logs MUST be retained for the duration of any applicable limitation period for legal claims, typically 3-6 years depending on jurisdiction.

Logging and monitoring data

Logs and monitoring data SHOULD be persisted in separate storage from an application’s primary data stores (for state and session data).

Personally identifiable information (PII) MUST NOT be sent to log output.

---

References

• NIST. Privacy Framework. National Institute of Standards and Technology. — A voluntary framework for managing privacy risk.

---

1. Taken from the Information and Privacy Commissioner of Ontario, Canada: https://www.ipc.on.ca/en/resources-and-decisions/privacy-design ↩