A Checklist for good regulation

Politics can help or hinder the development of evidence-based policy and regulation. Regulators are subject to tight timeframes and expectation to deliver “results”. As such, a policymaker may deliver the policy first, and the policy researcher finds evidence for the decision after the fact. Frequently the best a regulator can do is to “satisfice,” or do the minimum.  Of course, policymakers should do their best to incorporate evidence from the start and through the entire process of policymaking. The following checklist helps ensure that minimum standards are met. This checklist works for a variety of domains and polices.

• Aligned with national laws and institutional goals
• Based on rational, comprehensive data and evidence—both quantitative and qualitative
• Clearly states the reasons why it’s needed and the proposed outcome
• Provides a framework for achieving the outcome
• Concise, clearly communicated and widely understood
• Creates value and benefits with measurable outcomes
• Monitored, evaluated, and reviewed regularly.

Aligned with national laws and institutional goals

While the first requirement may seem obvious, it is surprising how frequently rules are adopted which do not conform to national laws, or at that the regulated parties believe they do not conform to the law. This could also suggest that laws are ambiguous, leaving the interpretation to the courts or to legislators to improve when updating statutes. As such, regulators should be sure that rules do not duplicate or conflict with existing laws and statutes.

Based on rational, comprehensive data and evidence—both quantitative and qualitative

While it may seem obvious that data and evidence should inform policy, it is helpful to review why.  The basic reason to include data and evidence is to justify why a policy will work or has worked. For example, the top 10 achievements in health in the United State were each preceded by some kind of evidentiary investigation, e.g. vaccines, prenatal screening, seat belt laws, limiting exposure to lead etc.

Telecommunications policy, like health, is based on a complex set of scientific, economic, legal, social, and political factors. All the same, there can be a gap between policy research and the policies that are enacted and enforced. More than $30 billion annually in spent in the US on health policy research, but health policy may or may not bear resemblance to the empirical research. A review by Hartsfield et al. identified 107 model public health laws covering 16 topics, but only 7 laws were based on scientific information (e.g., research-based guidelines).

Policymakers should make a best efforts attempt to include data and evidence and state its limitations. Similarly, if they cannot include data and evidence, then they should disclose the shortcoming. 

Clearly state the reasons why it’s needed and the proposed outcome

Good regulation should clearly state the problem, the proposed action and the expected outcome. the reasons why it is needed and the expected outcome.

Sadly, the Federal Communications Commission’s Open Internet policies fall short on this parameter. The problem is not clearly defined  or demonstrated. For example, there should be documented, systematic abuse for which the proposed rules would ameliorate. Such a process would then lead to the expected outcome. However, that through line is not established. The FCC merely states that its regulation is needed to maintain the status quo of “openness.” This argumentation does not follow as the internet is already open and policed by the Federal Trade Commission (FTC).

Ideally the policy should be prepared in a way in which success can be measured, for example, more startups, or more investment. But the Order sidesteps such measurement which would add accountability and credibility to the policy and the FCC.

Provides a framework for achieving the outcome

A policy that is clearly articulated with a genuine problem, a viable solution, and a path to success allows regulators to make a durable framework for the undertaking.

Evidence-based policy is comprised of (1) a rational process of evaluation of policy possibilities; (2) the collection and evaluation of information, both quantitative and qualitative; and (3) the measurement of outcomes expected and observed.

Concise, clearly communicated and widely understood

It policy with a sound problem formulation, set of actions, and expected outcomes with associated measures become easier to communicate to stakeholders, and by the same token, becomes easier to understand and support. Notably following the evidenced-based process forces policymakers to focus on the problem and solution with the appropriate instrument to achieve the goal. Such a process also forces the distillation of the policy into concepts which are easily to explain and understand. 

Creates value and benefits with measurable outcomes

Importantly, policy and regulation require resources to develop and implement. Ideally such undertakings create legitimate value and benefit which can be measured. Measurement is important, but it can be difficult and challenging as explained in the next section. But policymakers should not shirk this responsibility, and indeed stakeholders can also use measurement to hold policymakers and regulators accountable.

Monitored, evaluated, and reviewed regularly.

A telltale sign of a flawed policy is that it lacks measurable outcomes and hence allows policymakers and regulators to abdicate responsibility. However, stakeholders should press for the proper measurement and evaluation. By the same token, monitoring and measuring can also take the form of “going through the motions” to restate meaningless information to create the appearance of legitimacy. Again, stakeholders should return to the policy objective and the problem and press for accountability.

Here are some common reasons for failure to collect information to make evidenced-based policy and regulation.

• Insufficient evidence base: scientific evidence is lacking or changing over time; the cost of collect and analyze information is high
• Mismatched time horizons: Election cycles, policy processes, and research time run on different schedules and are not coordinated
• Power of special interests: Different groups have different agenda which may or may not cohere with an evidence-based process
• Researchers disconnected from the policy process: While there are can be reasons to keep research and policy separate, the lack of communication means that research is developed and communicated in a way that makes it difficult for policymakers to action
• Research silos: while some research may be conducted properly, it lacks multidisciplinary perspective to give it nuance and depth
• Policymaking is a non-linear, ad hoc process:  Evidence-based policy occurs in complex systems and social psychology suggests that decision-makers often rely on habit, stereotypes, and cultural norms for the vast majority of decisions.
• Policymaker bias. Some may object to the idea of “measuring” the effectiveness of a policy. Indeed, an insistence on evidence is itself a bias, but one that is enshrined in most legal systems to ensure justice. If anything, measurement can be a way to provide additional support and justification for desired policies. For example, there are indices of freedom[1] and human rights,[1] and these are valuable to make policy comparisons across countries.