Gov't officials' ambiguous policy answers in Diet prove troublesome for AI analysis

Understanding the ambiguity of Cabinet and other government officials' answers in Diet deliberations were found to be beyond the understanding of artificial intelligence (AI) tasked with drafting their answers, an experiment by the Ministry of Economy, Trade and Industry (METI) has found.

The experiment was conducted by a consulting firm contracted by the ministry for 18 million yen. The company's AI language processing service was fed with the last five years of Diet records, and when a possible question posed by a lawmaker is input into the system, the AI is supposed to present similar questions from past deliberations and the main points of draft model responses to that question based on past answers recorded in the minutes. The process was envisioned to make preparing responses to Diet questions at the ministry more efficient, as it can sometimes take all night for such sessions.

To test the AI's ability, roughly 80 employees used this system for two weeks in February 2017, and 50 of those employees answered a questionnaire at the end of the test period. Combining those who answered "not at all" and "not really," 48 percent responded that the AI did not produce the similar past questions nor the draft model responses based on past answers that matched the question the employee had input into the system.

When the results were analyzed, the AI failed to recognize policy terminology such as "corporate tax rate" and "manufacturing subsidy," and instead focused on general terms like "efforts" and "smooth," thus reducing the accuracy of the search for similar questions. As there were also cases where the viability of implementing policies was not clearly answered in the data, the AI also experienced difficulties in deciding how it should respond to questions.

The METI then investigated the possibility of using the system after organizing response documents created by every department in the ministry, the minister's answers at press conferences and more in the AI's database in order to provide information necessary to respond to questions.

"In the Diet, there are all sorts of conversations that involve 'bargaining,' 'making things ambiguous,' 'evasion' and other tactics," explains a METI representative. "A high level of linguistic judgment is necessary."

An official from one of the central ministries and agencies commented, "There is a special technique to make it seem like you are answering a question when you are actually dodging it. I think that would be difficult for the AI to process," doubtful that the AI will be suited to the given task. Another official pointed out, "Differences between the ruling and opposition parties delay the decision of Diet schedules, and the questions come the evening before or in the late hour of the night. That's the real problem. If that isn't fixed, then there is no way to make work more efficient."

"Eventually it will be possible to create a system that is both accurate and clear," says Constitutional Democratic Party of Japan lawmaker Kazuma Nakatani, 34, who has been involved in the founding of an IT company. "Once that happens, the Cabinet members and bureaucrats that have given ambiguous answers in the Diet will be confronted by the AI's clear answers. I hope that it will encourage them to re-examine the way they answer questions."