Improving Australia’s data system

Key points

Sourced from chapter 9 and 10 of the Productivity Commission's Final Report

Implementation processes for Australia’s new data framework must be as open and transparent as possible. All beneficiaries — the community, governments and businesses — should be involved in an exchange of views around data handling and use, in a manner that raises confidence in Australia’s data ecosystem. Changes of this order cannot be suddenly launched into the public arena.

Commencing consultations on improving data access in line with community expectations and releasing non-sensitive public sector data should be a priority for governments. But achieving the large-scale change needed to position Australia to make the most of its data now and in the years ahead will require time, detailed planning and strong leadership from the highest levels of government.

The Commission has made a number of recommendations that can be implemented in the short term to improve data availability and use.

Streamlining data access and eliminating inefficiencies

  • Abolishing the requirements to destroy linked datasets and linkage keys at the completion of projects that include Commonwealth data.
  • Accrediting state-based linkage units to link Commonwealth data.
  • Commencing discussions with the National Health and Medical Research Council and research institutions on the recommended reforms to the operation of human research ethics committees, and drafting an intergovernmental agreement on mutual recognition for ethics approvals and streamlining governance approvals.
  • Altering existing Department of Finance contract templates for procurement (and equivalent templates at the state and territory level) to incorporate clauses that would vest access, or at least purchase rights, to data in governments on a default basis. This would ensure that the interests of the public, with respect to data, are upheld in all private-public contract negotiations.

Improving data availability

  • Creating and publishing registers of data (including metadata and linked datasets) held by publicly funded entities. This includes all Australian Government agencies and other bodies that fund research, such as the Australian Research Council and the National Health and Medical Research Council.
  • Changes to government contracting will ensure access to publicly-funded data collected by the private sector can be made without the need to update legislation.

Setting up the new Data Framework

  • Establishing the office of the NDC, and tasking it with developing guidance on de-identification and processes for the accreditation of release authorities and trusted users.
  • Conducting extensive consultation on the comprehensive right, and drafting the Data Sharing and Release Act.

Pricing public sector data

There are various approaches for pricing public sector data, ranging from free provision and marginal cost pricing to commercial pricing. Which approach is most suitable will vary according to user demand and agency capability to act commercially.

For a given level of data quality, making data freely available in a timely way will maximise use and therefore deliver the highest level of social benefits. However, it will increase the net cost to government of data release.

Where agencies undertake substantial transformation because it meets the principles above, there are strong grounds for passing these additional costs on to data users.

An exception may be where data is used for research in the public interest. Pricing of data for the publicly funded research community should be the subject of a separate review.

Please send comments or enquiries to the Taskforce via

Productivity Commission Recommendations

6.1, 6.2, 6.3, 6.4, 6.5, 6.14, 6.15, 6.17, 8.4, 9.1, 9.2, 9.3, 10.1, 10.3 and 10.4

Recommendation Description

As an immediate objective, all Australian governments should direct the early release of all non-sensitive publicly funded datasets — whether held by a government agency or other body receiving public funding for data collection activities.

A realistic assessment of the risks attached to public release of identifiable information that is already public (in a less accessible form) should be undertaken by all governments, with the intention of releasing low risk data, and mitigating risks where possible to enable far greater public release of data, including that which could be used for program or agency performance management purposes.

Agencies should report annually on the proportions of their datasets made publicly available, shared, and not available for release.


Additional qualified entities should be accredited to undertake data linkage.

State-based data linkage units should be able to apply for accreditation by the National Data Custodian (recommendation 6.6) to allow them to link Australian Government data.


All Australian governments entering into contracts with the private sector that involve the creation of datasets in the course of delivering public services should assess the strategic significance and public interest value of the data as part of the contracting process.

Where data is assessed to be valuable, governments should retain the right to access or purchase that data in machine-readable form and to subsequently apply any analysis and release strategy that is in the public interest.

The Australian Government Department of Finance should modify template contracts to, by default, vest access and purchase rights in governments, and avoid the need for negotiating separate rights in each contract. State and Territory governments should adopt a similar approach.


Publicly funded entities, including all Australian Government agencies, should create comprehensive, easy to access registers of data, including metadata and linked datasets, that they fund or hold. These registers should be published on Where datasets are held or funded but are not available for access or release, the register should indicate this and the reasons why this is so.

States and Territories should create an equivalent model for their agencies where such registers do not exist. These should, in turn, be linked to

A reasonable timeframe in which to achieve this is within one year (by March 2018).


In determining datasets for public release, a central government agency in each jurisdiction with overarching policy responsibility for data should offer a public process whereby datasets or combinations of datasets can be nominated, with a public interest case made, for release.

A list of requested datasets, and decisions regarding dataset release or otherwise, should be transparent and published online — in the Commonwealth’s case, on


Progress by individual research institutions receiving Australian Government funding in making their unique research data and metadata widely available to others should be openly published by those institutions, with reference to past performance.

All bodies channeling public funds for research, such as the National Health and Medical Research Council and Australian Research Council, should similarly require in future funding agreements with research applicants that data and metadata is to be publicly available, and publish the results of progress on this for their funded projects.

On completion of projects, research institutions should include in their reports details of when and how other researchers can access the project’s data and metadata.


Processes for obtaining approval from human research ethics committees (HRECs) should be streamlined.

To achieve this in the health sector:

  • All HRECs should be required to register with the National Health and Medical Research Council (NHMRC). The NHMRC should receive funding to expand its current registration process, to include audits of registered HRECs.
  • To maintain their registration, HRECs must implement efficient and timely approval processes, which ensure projects are not unduly delayed. The time taken to consider and review projects should be reported to the NHMRC, and included in the annual report on HREC activity.
  • As a condition for registration, all HRECs and the institutions they operate in would be required to accept approvals issued by certified HRECs for multi-site projects, without additional reviews. The Australian Health Ethics Committee should develop uniform review processes to be used by certified HRECs.
  • The Council of Australian Governments’ Health Council should sign an intergovernmental agreement that extends the existing National Mutual Acceptance Scheme to all jurisdictions, including the Commonwealth, and all types of projects. As part of this agreement, all jurisdictions should also implement streamlined governance approvals.

The Australian Government should abolish its requirement to destroy linked datasets and statistical linkage keys at the completion of researchers’ data integration projects. Where an Accredited Release Authority is undertaking multiple linkage projects, it should work towards creating enduring linkage systems to increase the efficiency of linkage processes.

Data custodians should be advised as part of early implementation of this reform package to use a risk-based approach to determine how to enable ongoing use of linked datasets. The value added to original datasets by researchers should be retained and made available to other dataset users.


The Australian Government’s Protective Security Policy Framework (and equivalent State and Territory policies) should be amended to recognise that the risk and therefore the classification needed for data can be reduced by:

  • Transforming a dataset, for example through de-identification, such that the risks of misuse on dataset release are reduced.
  • Only making the transformed data available to trusted researchers in a secure computing environment, with usage monitored and output checked for disclosiveness.

This would align the Protective Security Policy Framework with the current legal environment.

The Australian Government should consider doing this as part of its response to the Belcher Review.


The emphasis for government agencies in handling data should be on making data available at a ‘fit for release’ standard in a timely manner. Beyond this, agencies should only transform data beyond the basic level if there is a clearly identified public interest purpose or legislative requirement for the agency to undertake additional transformation, or:

  • the agency can perform the transformation more efficiently than either any private sector entities or end users of the data
  • users have a demonstrable willingness to pay for the value added product
  • the agency has the capability and capacity in-house or under existing contract
  • the information technology upgrade risk is assessed and found to be small.

The pricing of public sector datasets for public interest research purposes should be the subject of an independent review.


Minimally processed public sector datasets should be made freely available or priced at marginal cost of release.

Where data has been transformed, the transformed dataset may be priced above the marginal cost of release. Data custodians should experiment with low prices initially to gauge the price sensitivity of demand, with a view to sustaining lower prices if demand proves to be reasonably price sensitive.


The Australian Government should engage actively with the community on matters related to data availability and use.

At a minimum, the National Data Custodian should regularly convene forums for consultation, to ensure community concerns about increased use of data are addressed.


Government agencies should adopt and implement data management standards to support increased data availability and use as part of their implementation of the Australian Government’s Public Data Policy Statement.

These standards should:

  • be published on agency websites
  • be adopted in consultation with data users and draw on existing standards where feasible
  • deal effectively with sector-specific differences in data collection and use
  • support the sharing of data across Australian governments and agencies
  • enable all digitally collected data and metadata to be available in commonly used machine-readable formats (that are relevant to the function or field in which the data was collected or would likely be most commonly used), including where relevant and authorised, for machine-to-machine interaction.

Policy documents outlining the standards and how they would be implemented should be available in draft form for consultation by the end of 2017, with standards implemented by the end of 2020.

Agencies that do not adopt agreed sector-specific standards would be noted as not fully implementing the Australian Government’s Public Data Policy and would be required to work under a nominated Accredited Release Authority to improve the quality of their data holdings.


The private sector is likely to be best placed to determine sector-specific standards for data sharing between firms, where required by reforms recommended under the new data Framework.

In the event that cooperative approaches to determining standards and data quality do not emerge or adequately enable data access and transfer (including where sought by consumers), governments should facilitate this.