HomeArticlesServicePriceAbout

"Large Language Models for Semantic Monitoring of Corporate Disclosures: A Case Study on Korea’s Top 50 KOSPI Companies"
This paper explores the feasibility of automating sentiment analysis of corporate disclosures from Korea’s top 50 KOSPI-listed companies using large language models (LLMs) such as OpenAI’s GPT-3.5-turbo and GPT-4.

1. Key Objectives

• Assess the effectiveness of LLMs in analyzing the sentiment of corporate disclosures.

• Compare the performance of GPT-3.5-turbo and GPT-4 models.

• Identify challenges and limitations in sentiment analysis using LLMs.

2. Research Methodology

• The top 50 KOSPI-listed companies were selected as of June 28, 2023.

• Monthly summaries of corporate disclosures from January 1, 2022, to May 31, 2023, were collected.

• GPT models were instructed to evaluate key factors such as financial stability, market share, and growth potential.

• Sentiment assessments generated by GPT models and financial experts were compared using a scale from 1 (highly negative) to 5 (highly positive).

• Inter-rater agreement was measured using Cohen’s Kappa statistic and simple agreement rate.

• The correlation between GPT models and human evaluations was analyzed using the Spearman correlation coefficient and Kendall rank correlation coefficient.

3. Key Findings

• Superior Performance of GPT-4: GPT-4 exhibited a stronger correlation with human evaluations than GPT-3.5-turbo in all conditions, indicating its superior ability to comprehend and assess complex financial language.

• Optimal Condition: GPT-4 achieved the highest performance across all evaluation criteria under Condition 2, which applied adjustments to reduce overly positive sentiment assessments.

• Sentiment Bias: The study found that GPT models tended to overestimate the positivity of corporate disclosures, suggesting a natural inclination toward interpreting content favorably. This bias needs to be mitigated through further refinements. However, the study also highlights that these limitations can be addressed through additional adjustments.

• Agreement with Expert Opinions: Cohen’s Kappa statistic was recorded at 0.352, and the simple agreement rate stood at 68%, indicating a relatively reasonable level of agreement with expert assessments.

4. Limitations & Challenges

• LLMs lack background knowledge of the analyzed companies, limiting the depth and contextual understanding of their analysis.

• They are unable to incorporate external data sources such as financial data tables or news articles.

• They have limited capability in understanding and executing complex financial formulas or advanced statistical analysis.

• LLM performance may fluctuate over time, leading to inconsistencies.

This study demonstrates that LLMs, particularly GPT-4, have significant potential in conducting sentiment analysis on Korean corporate disclosures. However, further research is needed to address the limitations of LLMs, mitigate biases, enhance contextual understanding, integrate external data sources, and improve mathematical analysis capabilities. This research contributes to a better understanding of LLMs’ capabilities and limitations in real-time sentiment monitoring, providing valuable insights for both academia and industry practitioners.

[Compliance Note]

• All posts by Sellsmart are for informational purposes only. Final investment decisions should be made with careful judgment and at the investor’s own risk.

• The content of this post may be inaccurate, and any profits or losses resulting from trades are solely the responsibility of the investor.

• Core16 may hold positions in the stocks mentioned in this post and may buy or sell them at any time.