This is an old version of an article published in Plural’s blog. You can find the new version here.
So far in 2023, thousands of bills have been introduced in jurisdictions across the country. The sooner people understand them, the sooner they can start advocating for what they care about. Since bills can be long and complex, it’s often helpful to have a summary of the changes the bill is proposing.
Let’s take SB 396:
“TO CREATE THE SOCIAL MEDIA SAFETY ACT; TO REQUIRE AGE VERIFICATION FOR USE OF SOCIAL MEDIA; AND TO CLARIFY LIABILITY FOR FAILURE TO PERFORM AGE VERIFICATION FOR USE OF SOCIAL MEDIA AND ILLEGAL RETENTION OF DATA.”.
This bill, from Arkansas’s 2023 Session, regulates minors’ use of social media, like verifying users’ age and requiring minors to have parental consent. The bill is pages long with technical jargon and multiple textual changes. In this case, as for most Arkansas bills, the release does not include a legislature provided summary.
Given the advancements of Artificial Intelligence in language tasks, we believe Language Models can help us understand bills. Here’s an AI summary for SB 396:
“In SB 396, the bill proposes the creation of the Social Media Safety Act, which requires social media companies to verify the age of account holders, obtain parental consent for minors, and use reasonable age verification methods, which may include obtaining digitized identification or using other commercially reasonable age verification methods. The bill specifies that social media companies that knowingly violate the proposed act will be liable for damages resulting from a minor accessing a social media platform. The bill also outlines liability for commercial entities and third-party vendors related to the retention of identifying information after access to a social media platform has been granted.”
Summarizing Bill Texts
Language Models (LM) are specific generative models trained with big amounts of unstructured text data, with the goal of predicting the next word or the words adjacent inside the text corpus. With enough training data, number of parameters and computing power, these models can perform language tasks at human level and even maintain logical conversations. Language models are at the base of technologies such as ChatGPT, Bard, and others.
Given the current capabilities of LM and generative AI, we decided to try using it to summarize bills. There’s ample evidence of using LLM to summarize new articles, emails, contracts, and even short stories. Longer texts such as books still present a challenge. Given the success in other fields, we decided to analyze bill texts using common summarization techniques.
But it’s not a trivial task! Our first round of testing revealed some barriers and rough edges to this approach. Here are a few examples of our first round of AI generated summaries, with corresponding Policy Expert Evaluations:
| Bill Name | AI Model-Generated Summary | Policy Expert Evaluation |
| SF 13: Juneteenth (June 19) state holiday recognition - Minnesota 2023-2024 REGULAR SESSION | This bill recognizes Juneteenth, June 19th, as a state holiday and amends Minnesota Statutes 2022, sections 10.55 and 645.44, subdivision 5. It also requires the governor to issue a proclamation honoring the observance and recognizing the contributions African-Americans have made to Minnesota’s communities, culture, and economy. Additionally, public schools may offer instruction and programs on the occasion. The bill also states that when Juneteenth falls on a Sunday or Saturday, the following or preceding day shall be a holiday. | Capturing the main intent of the bill, establishing Juneteenth as a state holiday, is quite impressive given that this would be a difficult bill for a layperson to understand. However, the summary indicates that the bill creates new requirements for the governor to issue a proclamation and encourages public school instruction on Juneteenth. In this, the summary is incorrect as that text already existed in statute and is not newly added by this legislation. |
| SF 23: Conversion therapy prohibition establishment - Minnesota 2023-2024 REGULAR SESSION | This bill prohibits conversion therapy with children or vulnerable adults, prohibits medical assistance coverage for conversion therapy, and prohibits the misrepresentation of conversion therapy services or products. It defines conversion therapy as any practice by a mental health practitioner or professional that seeks to change an individual’s sexual orientation or gender identity, and prohibits the use of any fraud, false pretense, false promise, false guarantee, misrepresentation, false or misleading statements, or deceptive practice when advertising or offering conversion therapy services. | This summary does an excellent job of concisely summarizing the bills main actions and the details that really matter, like how these terms are defined. |
| SF 15: Full Social Security benefit subtraction and public pension benefit subtraction establishment - Minnesota 2023-2024 REGULAR SESSION | This bill amends Minnesota Statutes 2022 to establish a public pension benefit subtraction for individual income taxation. It sets out the maximum subtraction and threshold amounts for married taxpayers filing a joint return, single or head-of-household taxpayers, and married taxpayers filing separate returns. It also defines provisional income, public pension exclusion, qualified benefits, and the maximum state exclusion. It also amends Minnesota Statutes 2022, section 290.091, subdivision 2, to define alternative minimum taxable income, investment interest, net minimum tax, regular tax, and tentative minimum tax. This section is effective for taxable years beginning after December 31, 2022. | The summary doesn’t include the bill’s proposed removal of the cap on social security income that a taxpayer can subtract, which is a huge part of the impact of the bill. The latter half of the summary, starting with “it also defines provisional”, is also inaccurate as the bill itself makes no changes to law here. The model is interpreting existing statute text as if it is what the bill is proposing. |
As you can see, while the model is able to summarize some bills successfully, there are other cases where the model confuses statute and bill text, giving an incorrect summary of the proposed bill.
Summarizing Bill Changes
Using AI and LM to summarize bills may be more complex than it seems. Since bills are mainly proposed changes to current statutes, if we pass the proposed text to a summarization model, it is often not able to differentiate between current and proposed law. However, if we pass only the changes in text, the model won’t have the necessary context to understand what the bill is proposing to change — for instance, modifying the minimum wage from $12 to $15 may be a single character change.
After additional research and help from our policy experts, we were able to find a combination of models and prompts that correctly identify most of the changes, with a better understanding of the current statutes and proposed law. Although it’s not perfect, the performance is significantly better.
| Bill Name | Model Summary | Policy Expert Evaluation |
| SF 13: Juneteenth (June 19) state holiday recognition - Minnesota 2023-2024 REGULAR SESSION | SF 13 proposes to amend Minnesota Statutes 2022, sections 10.55 and 645.44, subdivision 5, to recognize Juneteenth, June 19, as a state holiday in Minnesota. It specifies that Juneteenth is to be observed annually on the third Saturday in June and includes Juneteenth as a holiday in the definition of “Holiday” along with other nationally recognized holidays. The bill also mandates the governor to issue a proclamation honoring the observance and recognizing the contributions made by African-Americans to Minnesota’s communities, culture, and economy. | This is a really hard one to get, but it does a pretty good job. Ideally, it would indicate that the first section of the bill changes Juneteenth from being recognized on the third Saturday, to being recognized on June 19th. |
| SF 23: Conversion therapy prohibition establishment - Minnesota 2023-2024 REGULAR SESSION | The bill proposed in SF 23 aims to prohibit conversion therapy with minors and vulnerable adults, prohibit medical assistance coverage for conversion therapy, and prohibit the misrepresentation of conversion therapy services or products. It also proposes amending Minnesota Statutes to add a new law to protect individuals from conversion therapy and to add definitions for conversion therapy and related terms, as well as outlining penalties for those who engage in or advertise conversion therapy. | Very good again. 100% accurate. |
| SF 15: Full Social Security benefit subtraction and public pension benefit subtraction establishment - Minnesota 2023-2024 REGULAR SESSION | SF 15 is proposing an amendment to Minnesota Statutes related to taxation and individual income. The bill proposes to establish a public pension benefit subtraction and make amendments to sections 290.0132 and 290.091 of the Minnesota Statutes.The bill would add a new subdivision 31 to section 290.0132 to establish a public pension income subtraction. The subtraction would be determined by the sum of the federal and state exclusions, with the maximum state exclusion being $5,450 for a joint return, half that amount for a married taxpayer filing a separate return, and $4,260 for all other taxpayers.The bill would also amend subdivision 26 of section 290.0132 related to Social Security benefits. It would allow a portion of taxable Social Security benefits received by a taxpayer in a taxable year as a subtraction. The maximum subtraction would be determined based on the taxpayer’s filing status and provisional income.Finally, the bill would amend subdivision 2 of section 290.091 related to the definitions of terms used in the tax imposed by that section. | This summarization is accurate and valuable. There are some things that are slightly incorrect — for example, in paragraph 3, it says “it would allow a portion of taxable social….”, when in fact the bill allows all social security taxes to be subtracted, not a portion. |
Next steps
Despite improved performance, there are still hurdles to overcome. LMs can hallucinate, confidently share wrong answers, and even give biased responses. LMs also have a hard time analyzing very long texts, so large omnibus bills may be out of the question for now. Finally, there’s a lot of room for improvement in the model and prompting area besides this experiment.
While limitations exist, it’s likely that LMs and Generative AI can support and improve policy analysis, as it has been shown in other text based fields. While this remains a hard problem, we’re committed to keep exploring the edges of the intersection between AI and Public Policy.