Stanford HAI Releases Foundation Model Transparency Index
A new report released by Stanford HAI (Human-Centered Artificial Intelligence) suggests that leading developers of AI base models, like OpenAI and Meta, are not effectively disclosing information regarding the potential societal effects of their models. The Foundation Model Transparency Index, unveiled today by Stanford HAI, evaluated the transparency measures taken by the makers of the top 10 AI models. While Metaβs Llama 2 ranked the highest, with BloomZ and OpenAIβs GPT-4 following closely behind, none of the models achieved a satisfactory rating.
LOOI Robot-Space Black-AI Desktop Companion,ChatGPT&Gemini Voice Interaction,Visual Understanding,Personality&Memory,10W Wireless Charging,Tech&Gadget Gift,Electronic pets,Toy,Geek,Desktop Setup
1. Smarter Conversations, Powered by ChatGPT π€ LOOI brings natural, intelligent conversations to your desk with ChatGPT voice...
As an affiliate, we earn on qualifying purchases.
Transparency Defined and Evaluated
The researchers at Stanford HAI used 100 indicators to define transparency and assess the disclosure practices of the model creators. They examined publicly available information about the models, focusing on how they are built, how they work, and how people use them. The evaluation considered whether companies disclosed partners and third-party developers, whether customers were informed about the use of private information, and other relevant factors.
Anki Vector 2.0 AI ChatGPT Connected Robot Companion β Smart Autonomous Home Robot with Face Recognition and Voice Conversations - ChatGPT Subscription Required (Black)
AI-Powered & Fully Autonomous: Vector navigates, recognizes faces, and reacts to his surroundings with lifelike independence β no...
As an affiliate, we earn on qualifying purchases.
Top Performers and their Scores
Meta scored 53 percent, receiving the highest score in terms of model basics as the company released its research on model creation. BloomZ, an open-source model, closely followed at 50 percent, and GPT-4 scored 47 percent. Despite OpenAIβs relatively closed design approach, GPT-4 tied with Stabilityβs Stable Diffusion, which had a more locked-down design.
Pocket AI Voice Recorder & Smart Assistant β Auto Transcription, Summaries & Action Items β AI Note Taker for Meetings, Calls & Productivity - Space Grey
YOUR AI PERSONAL ASSISTANT FOR EVERYDAY PRODUCTIVITY: More than a voice recorder, Pocket works as your AI personal...
As an affiliate, we earn on qualifying purchases.
OpenAIβs Disclosure Challenges
OpenAI, known for its reluctance to release research and disclose data sources, still managed to rank high due to the abundance of available information about its partners. The company collaborates with various companies that integrate GPT-4 into their products, resulting in a wealth of publicly available details.
EarlySincere 4K AI Smart Glasses with Photochromic Lenses, Integrated ChatGPT AI Model, Real-Time Translation, HD Video Recording, Long Battery Life, Bluetooth 5.4 Wireless
πγππ πππππ πππππππππ πππππππ ππππ πππγ This advanced wearable camera empowers you to capture the wonders of the...
As an affiliate, we earn on qualifying purchases.
Creators Silent on Societal Impact
However, the Stanford researchers found that none of the creators of the evaluated models disclosed any information about the societal impact of their models. There is no mention of where to direct privacy, copyright, or bias complaints.
Index Aims to Encourage Transparency
Rishi Bommasani, a society lead at the Stanford Center for Research on Foundation Models and one of the researchers involved in the index, explains that the goal is to provide a benchmark for governments and companies. Proposed regulations, such as the EUβs AI Act, may soon require developers of large foundation models to provide transparency reports. The index aims to make models more transparent by breaking down the concept into measurable factors. The group focused on evaluating one model per company to facilitate comparisons.
OpenAIβs Research Distribution Policy
OpenAI, despite its name, no longer shares its research or codes publicly, citing concerns about competitiveness and safety. This approach contrasts with the large and vocal open-source community within the generative AI field.
The Verge reached out to Meta, OpenAI, Stability, Google, and Anthropic for comments but has not received a response yet.
Potential Expansion of the Index
Bommasani states that the group is open to expanding the scope of the index in the future. However, for now, they will focus on the 10 foundation models that have already been evaluated.
