An issue in langchain v.0.0.171 allows a remote attacker to execute arbitrary code via the via the a json file to the load_prompt parameter.

**langchain vulnerable to arbitrary code execution**

Critical severity GitHub Reviewed Published Aug 22, 2023 to the GitHub Advisory Database • Updated Aug 23, 2023

GHSA-7gfq-f96f-g85j: langchain vulnerable to arbitrary code execution

By Habiba Rashid
The campaign, which began at the start of August 2023, revolves around malicious packages impersonating the legitimate noblox.js,…
This is a post from HackRead.com Read the original post: Luna Grabber Malware Hits Roblox Devs Through npm Packages

The campaign, which began at the start of August 2023, revolves around malicious packages impersonating the legitimate noblox.js, a popular Node.js Roblox API wrapper.

*   **Roblox developers are being targeted by a new malware called Luna Grabber**

*   **The malware is being distributed through malicious npm packages that impersonate legitimate software.**

*   **Luna Grabber is capable of stealing sensitive data from victims’ web browsers, Discord applications, and local system configurations.**

*   **The malware was downloaded approximately 1000 times, but its impact was relatively low due to the security measures in place to protect developers on the npm repository.**

*   **The incident highlights the growing trend of malicious actors employing typo squatting to exploit developers’ trust in legitimate software packages.**

Cybersecurity firm ReversingLabs has uncovered a sophisticated cyber attack targeting developers on the Roblox gaming platform. Malicious actors have been distributing malicious packages through the npm public repository, attempting to exploit users by mimicking legitimate software while incorporating malicious payloads that steal sensitive information from victims’ systems.

**Malware Campaign Overview**

The campaign, which began at the start of August 2023, revolves around malicious packages impersonating the legitimate noblox.js, a popular Node.js Roblox API wrapper. By infiltrating the npm public repository, attackers capitalized on unsuspecting developers seeking to interact with the Roblox gaming platform using scripts.

ReversingLabs researchers identified several malicious packages during the campaign, including noblox.js-vps, noblox.js-ssh, and noblox.js-secure. These packages were engineered to deliver multi-stage malicious payloads that targeted victims’ local web browsers and Discord applications. The most notable payload identified was Luna Grabber, an open-source malware designed to extract sensitive data.

**Malware Execution and Strategy**

Attackers meticulously designed the malicious packages to closely resemble the legitimate noblox.js package. By mirroring the original code and adopting similar naming conventions, the attackers aimed to deceive developers into downloading and using the compromised software.

The malicious packages leveraged a variety of techniques to compromise victims’ systems, including the incorporation of a separate file named postinstall.js. This post-installation script triggered the execution of a malicious payload after the package installation was completed. The malware then determined whether the victim was operating a Windows machine and proceeded to download and execute the Luna Grabber malware from Discord’s Content Delivery Network (CDN).

**Luna Grabber: Information Stealing Malware**

The research revealed that the primary payload of the malicious packages was Luna Grabber, a highly customizable malware capable of stealing information from victims’ web browsers, Discord applications, and local system configurations. The malware was also equipped with features that enabled it to detect virtual environments and initiate a self-destruct mechanism if necessary.

Interestingly, the attackers behind the campaign took advantage of the user-friendly nature of Luna Grabber’s builder application, simplifying the process of creating and configuring the malicious executable.

Screenshot shared by ReversingLabs shows Luna Grabber builder

While Luna Grabber’s open-source nature allowed attackers to tailor the malware to their needs, the choice of targeting developers on the Roblox platform suggests a focus on a specific user group.

**Limited Impact and Lessons Learned**

Despite the campaign’s sophistication, its impact remained relatively low. The malicious packages were downloaded approximately 1000 times, signalling that the security measures in place to protect developers on the npm repository were successful in limiting the reach of the attack.

The incident sheds light on the growing trend of malicious actors employing typo squatting to exploit developers’ trust in legitimate software packages. This approach has been previously observed in other campaigns, such as the IconBurst and Brainleeches campaigns.

Fake noblox.js-ssh’s npm website page (ReversingLabs)

While multi-stage malicious packages are common on certain open-source platforms, such as PyPI, their presence on npm—where this campaign took place—represents the ongoing challenge of maintaining secure open-source repositories and the importance of cautiousness in choosing software packages for development purposes.

1.  CISA warns of trojanized JavaScript library’s NPM package
2.  Discord.io Admits Data Breach: Info of 760K Users Sold Online
3.  6 official Python repositories plagued with cryptomining malware

Luna Grabber Malware Hits Roblox Devs Through npm Packages

An issue was discovered in json-c through 0.15-20200726. A stack-buffer-overflow exists in the function parseit located in json_parse.c. It allows an attacker to cause code Execution.

cmake ..-DCMAKE\_CXX\_FLAGS="-fsanitize=address -g" -DCMAKE\_C\_FLAGS="-fsanitize=address -g" -DCMAKE\_EXE\_LINKER\_FLAGS="-fsanitize=address"

    =================================================================
    ==12668==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffea9c409b0 at pc 0x00000051747b bp 0x7ffea9c388b0 sp 0x7ffea9c388a8
    READ of size 1 at 0x7ffea9c409b0 thread T0
        #0 0x51747a in parseit /home/seviezhou/jsonc/apps/json_parse.c:89:44
        #1 0x51747a in main /home/seviezhou/jsonc/apps/json_parse.c:182
        #2 0x7fc02a77783f in __libc_start_main /build/glibc-e6zv40/glibc-2.23/csu/../csu/libc-start.c:291
        #3 0x41a928 in _start (/home/seviezhou/jsonc/build/apps/json_parse+0x41a928)
    
    Address 0x7ffea9c409b0 is located in stack of thread T0 at offset 33008 in frame
        #0 0x51651f in main /home/seviezhou/jsonc/apps/json_parse.c:159
    
      This frame has 3 object(s):
        [32, 176) 'rusage.i.i.i' (line 42)
        [240, 33008) 'buf.i' (line 52) <== Memory access at offset 33008 overflows this variable
        [33264, 33408) 'rusage.i' (line 42)
    HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
          (longjmp and C++ exceptions *are* supported)
    SUMMARY: AddressSanitizer: stack-buffer-overflow /home/seviezhou/jsonc/apps/json_parse.c:89:44 in parseit
    Shadow bytes around the buggy address:
      0x1000553800e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x1000553800f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x100055380100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x100055380110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x100055380120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    =>0x100055380130: 00 00 00 00 00 00[f2]f2 f2 f2 f2 f2 f2 f2 f2 f2
      0x100055380140: f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2
      0x100055380150: f2 f2 f2 f2 f2 f2 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
      0x100055380160: f8 f8 f8 f8 f8 f8 f8 f8 f3 f3 f3 f3 f3 f3 f3 f3
      0x100055380170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x100055380180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    Shadow byte legend (one shadow byte represents 8 application bytes):
      Addressable:           00
      Partially addressable: 01 02 03 04 05 06 07
      Heap left redzone:       fa
      Freed heap region:       fd
      Stack left redzone:      f1
      Stack mid redzone:       f2
      Stack right redzone:     f3
      Stack after return:      f5
      Stack use after scope:   f8
      Global redzone:          f9
      Global init order:       f6
      Poisoned by user:        f7
      Container overflow:      fc
      Array cookie:            ac
      Intra object redzone:    bb
      ASan internal:           fe
      Left alloca redzone:     ca
      Right alloca redzone:    cb
    ==12668==ABORTING

CVE-2021-32292: A stack-buffer-overflow in json_parse.c:89:44 · Issue #654 · json-c/json-c

An issue was discovered in function ciMethodBlocks::make_block_at in Oracle JDK (HotSpot VM) 11, 17 and OpenJDK (HotSpot VM) 8, 11, 17, allows attackers to cause a denial of service. Note: Vendor states that this to is Defense in Depth at most due to the nature of the issue and the special circumstances required (server must be running particular code locally, code compiled with an old, old version of javac, etc.).

ADDITIONAL SYSTEM INFORMATION :  
OS version:  
Distributor ID: Ubuntu  
Description: Ubuntu 20.04.2 LTS  
Release: 20.04  
Codename: focal

JDK version we used:

openjdk version "17.0.2" 2022-01-18  
OpenJDK Runtime Environment (build 17.0.2+8-86)  
OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)

openjdk version "18" 2022-03-22  
OpenJDK Runtime Environment (build 18+36-2087)  
OpenJDK 64-Bit Server VM (build 18+36-2087, mixed mode, sharing)

openjdk version "19-ea" 2022-09-20  
OpenJDK Runtime Environment (build 19-ea+13-808)  
OpenJDK 64-Bit Server VM (build 19-ea+13-808, mixed mode, sharing)

A DESCRIPTION OF THE PROBLEM :  
When we run the test in jdk17.0.2, jdk18 and jdk19-ea, all in compiled mode(with "-Xcomp"), it crashed with the following message. But when run the test in mixed mode or interpreted mode(with "-Xint), it passed successfully.

The error message in compiled mode:

jdk17.0.2:  
\# A fatal error has been detected by the Java Runtime Environment:  
#  
\# SIGSEGV (0xb) at pc=0x00007f0b86b5a37b, pid=32365, tid=32379  
#  
\# JRE version: OpenJDK Runtime Environment (17.0.2+8) (build 17.0.2+8-86)  
\# Java VM: OpenJDK 64-Bit Server VM (17.0.2+8-86, compiled mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)  
\# Problematic frame:  
\# V \[libjvm.so+0x52837b\] ciMethodBlocks::make\_block\_at(int)+0x3b  
#  
\# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again  
#  
\# An error report file with more information is saved as:  
\# /home/minghai/hs\_err\_pid32365.log  
#  
\# Compiler replay data is saved as:  
\# /home/minghai/replay\_pid32365.log  
#  
\# If you would like to submit a bug report, please visit:  
\# https://bugreport.java.com/bugreport/crash.jsp

jdk18:  
\# A fatal error has been detected by the Java Runtime Environment:  
#  
\# SIGSEGV (0xb) at pc=0x00007f9a003d800b, pid=32250, tid=32263  
#  
\# JRE version: OpenJDK Runtime Environment (18.0+36) (build 18+36-2087)  
\# Java VM: OpenJDK 64-Bit Server VM (18+36-2087, compiled mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)  
\# Problematic frame:  
\# V \[libjvm.so+0x54b00b\] ciTypeFlow::get\_block\_for(int, ciTypeFlow::JsrSet\*, ciTypeFlow::CreateOption)+0x2b  
#  
\# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again  
#  
\# An error report file with more information is saved as:  
\# /home/minghai/hs\_err\_pid32250.log  
#  
\# Compiler replay data is saved as:  
\# /home/minghai/replay\_pid32250.log  
#  
\# If you would like to submit a bug report, please visit:  
\# https://bugreport.java.com/bugreport/crash.jsp

jdk19-ea:  
\# A fatal error has been detected by the Java Runtime Environment:  
#  
\# SIGSEGV (0xb) at pc=0x00007f96f291b1fb, pid=32319, tid=32332  
#  
\# JRE version: OpenJDK Runtime Environment (19.0+13) (build 19-ea+13-808)  
\# Java VM: OpenJDK 64-Bit Server VM (19-ea+13-808, compiled mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)  
\# Problematic frame:  
\# V \[libjvm.so+0x5571fb\] ciTypeFlow::get\_block\_for(int, ciTypeFlow::JsrSet\*, ciTypeFlow::CreateOption)+0x2b  
#  
\# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again  
#  
\# An error report file with more information is saved as:  
\# /home/minghai/hs\_err\_pid32319.log  
#  
\# Compiler replay data is saved as:  
\# /home/minghai/replay\_pid32319.log  
#  
\# If you would like to submit a bug report, please visit:  
\# https://bugreport.java.com/bugreport/crash.jsp

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :  
1\. extract the bug.zip  
2\. in dictionary "bug", run command:  
java -cp ./bugFiles:./util:./junit.jar:./hamcrest.jar:./target/classes:./target/test-classes org.junit.runner.JUnitCore com.alibaba.fastjson.deserializer.issue1463.TestIssue1463

you may add "-Xcomp" or "-Xint" to get different results.

EXPECTED VERSUS ACTUAL BEHAVIOR :  
EXPECTED -  
The result should be the same since the program are the same, no matter in compiled mode, mixed mode or interpreted mode.  
ACTUAL -  
When run in compiled mode(with "-Xcomp"), it crashed. But when run in mixed mode or interpreted mode(with -Xint), it passed successfully.

\---------- BEGIN SOURCE ----------  
to be attached in bug.zip  
\---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :  
run the command without -Xcomp or with -Xint

FREQUENCY : always

CVE-2022-40433: C2: segmentation fault in ciMethodBlocks::make_block_at(int)

webui-aria2 commit 4fe2e was discovered to contain a path traversal vulnerability.

CVE-2023-39141 is reserved for this vulnerability

Project link:

https://github.com/ziahamza/webui-aria2/

Vulnerability type:

Path traversal

Root cause: This line https://github.com/ziahamza/webui-aria2/blob/109903f0e2774cf948698cd95a01f77f33d7dd2c/node-server.js#L10 accepts file name from URL input, without sanitizing it to be in the same directory.

PoC:

When \`node-server.js\` is used, an attacker can simply request files outside the serving path

\`curl --path-as-is http://localhost:8888/../../../../../../../../../../../../../../../../../../../../etc/passwd\`

Root cause: Attacker may read any file that the www user can read.

Vulnerable versions:

Right now all versions even latest commit "109903f0e2774cf948698cd95a01f77f33d7dd2c" are vulnerable.

CVE-2023-39141: webui-aria2 CVE-2023-39141

An issue was discovered in Geomatika IsiGeo Web 6.0. It allows remote authenticated users to retrieve PHP files from the server via Local File Inclusion.

Passer au contenu

*   Géomatika
    *   Présentation
    *   Réalisations
    *   Géoservices à la carte.
    *   Applications métiers
    *   Recrutement
    *   Formations
    *   Contact
*   IsiGéo
    *   IsiGéo Web
    *   IsiGéo API Js
    *   IsiGéo Apps
    *   Portail cartographique
    *   Boite à outils
    *   IsiGéo – Développements en cours
*   Solutions
    *   Cadastre / Urbanisme
    *   Logiciel Cimetière pour les Mairies
    *   Logiciel d’adressage
    *   Catalogage de plans Autocad
    *   Gestion de l’ANC ( Assainissement Non Collectif )
    *   Contrôle Poteaux Incendie
    *   Gestion d’un parc d’éclairage public
*   Mobilité
    *   Solutions terrain
    *   GPS centimétriques pour les SIG
*   npg
*   

IsiGéo web Géomatika2023-01-23T22:44:57+01:00

Page load link

Aller en haut

CVE-2023-23565: IsiGéo web

Etcd v3.5.4 allows remote attackers to cause a denial of service via function PageWriter.write in pagewriter.go

fix(pkg/ioutil):avoid panic in PageWriter.Write() when pageBytes is 0

fix(pkg/ioutil): Trigger a panic when pageBytes is illegal in NewPateWriter

update(Test pkg/ioutil):Update TestPageWriterPageBytes

migrate e2e & integration role\_test to common

tests: Migrate Txn tests to common functional test framework

provide a generic assert function

server: Director can be stopped

Goroutine for new directors would live past director scope. Tests
could occassionally fail if this goroutine had log events after
test execution should have ended.

server: Add director interrupt handler

Director's goroutine would not be properly stopped in a non-test
scenario. Handler stops it when process is interrupted.

server: Move director interrupt handler to method

server: Don't register director interrupt handler

remove v2 http proxy in 3.6

tests: Extract cluster test cases

tests: Refactor spawn json command

hide the revision field when it isn't populated

tests: Make common framework context aware

Documentation: Publish v3.5 data inconsistency postmortem

scripts: Avoid additional repo clone

This PR removes additional clone when building artifacts.

When releasing v3.5.4 this clone was main cause of issues and
confusion about what release script is doing.

release.sh script already clones repo in /tmp/ directory, so clonning
before build is not needed. As precautions for bug in script leaving
/tmp/ clone in bad state  I moved "Verify the latest commit has the
version tag" and added "Verify the clean working tree" to be always run
before build.

scripts: Detect staged files before building release

fix a typo: print the correct error info

Governance: Use lazy consensus when needed to make decision

In lack of supermajority, we sometimes required to hold on to
important decisions for long time. In order to speed up, after
giving enough time for supermajority, use lazy consensus.

Encapsulation of applier logic: Move Txn related code out of applier.go.

The PR removes calls to applierV3base logic from server.go that is NOT part of 'application'.
The original idea was that read-only transaction and Range call shared logic with Apply,
so they can call appliers directly (but bypassing all 'corrupt', 'quota' and 'auth' wrappers).

This PR moves all the logic to a separate file (that later can become package on its own).

Encapsulating applier logic: UberApplier coordinates all appliers for server

This PR:
 - moves wrapping of appliers (due to Alarms) out of server.go into uber\_applier.go
 - clearly devides the application logic into: chain of:
     a) 'WrapApply' (generic logic across all the methods)
     b) dispatcher (translation of Apply into specific method like 'Put')
     c) chain of 'wrappers' of the specific methods (like Put).
 - when we do recovery (restore from snapshot) we create new instance of appliers.

The purpose is to make sure we control all the depencies of the apply process, i.e.
we can supply e.g. special instance of 'backend' to the application logic.

Marge applierV3Internal into applierV3 interface

Rename EtcdServer.Id with EtcdServer.MemberId.

It was misleading and error prone vs. ClusterId.

Applier does not depend on EtcdServer any longer.

All the depencies are explicily passed to the UberApplier factory method.

Move server/etcdserver/txn.go to new package: server/etcdserver/txn

Move etcdserver/errors.go to sepatate package to avoid cyclic dependencies.

Move apply to its own package (no dependency on etcdserver).

Apply encapsulation: Cleanup metrics reporting.

Side effect: applySec(0.4s) used to be reported as 0s, now it's correctly 0.4s.

Simplify imports and improve comments.

Move CheckTxnAuth to txn.

Rename package alising "apply2" -> apply.

Rename etcdserver/etcderrors package to etcdserver/errors.

expose UberApplier as interface (not as implementation struct).

Rename the txn, so as not to be the same as the package name.

Fixing missing comment on the dispatch() function.

Rename WrapApply to Apply.

Remove unused code and apply code-quality suggestions.

use go install instead of go get

feat(pkg/ioutil): verify.Assert is introduced into NewPageWritter

add etcd tool binaries into .gitignore

CVE-2022-34038: fix(pkg/ioutil):avoid panic in PageWriter.Write() when pageBytes is 0 by secsys-go · Pull Request #14022 · etcd-io/etcd

The json2xml package through 3.12.0 for Python allows an error in typecode decoding enabling a remote attack that can lead to an exception, causing a denial of service.

This section covers how to use the public PyPI download statistics dataset to learn more about downloads of a package (or packages) hosted on PyPI. For example, you can use it to discover the distribution of Python versions used to download a package.

Contents

*   Background
    
*   Public dataset
    
    *   Getting set up
        
    *   Data schema
        
    *   Useful queries
        
        *   Counting package downloads
            
        *   Package downloads over time
            
        *   Python versions over time
            
        *   Getting absolute links to artifacts
            
*   Caveats
    
*   Additional tools
    
    *   google-cloud-bigquery
        
    *   pypinfo
        
    *   pandas-gbq
        
*   References
    

**Background¶**

PyPI does not display download statistics for a number of reasons: 1

*   **Inefficient to make work with a Content Distribution Network (CDN):** Download statistics change constantly. Including them in project pages, which are heavily cached, would require invalidating the cache more often, and reduce the overall effectiveness of the cache.
    
*   **Highly inaccurate:** A number of things prevent the download counts from being accurate, some of which include:
    
    *   pip’s download cache (lowers download counts)
        
    *   Internal or unofficial mirrors (can both raise or lower download counts)
        
    *   Packages not hosted on PyPI (for comparisons sake)
        
    *   Unofficial scripts or attempts at download count inflation (raises download counts)
        
    *   Known historical data quality issues (lowers download counts)
        
*   **Not particularly useful:** Just because a project has been downloaded a lot doesn’t mean it’s good; Similarly just because a project hasn’t been downloaded a lot doesn’t mean it’s bad!
    

In short, because it’s value is low for various reasons, and the tradeoffs required to make it work are high, it has been not an effective use of limited resources.

**Public dataset¶**

As an alternative, the Linehaul project streams download logs from PyPI to Google BigQuery 2, where they are stored as a public dataset.

**Getting set up¶**

In order to use Google BigQuery to query the public PyPI download statistics dataset, you’ll need a Google account and to enable the BigQuery API on a Google Cloud Platform project. You can run up to 1TB of queries per month using the BigQuery free tier without a credit card

*   Navigate to the BigQuery web UI.
    
*   Create a new project.
    
*   Enable the BigQuery API.
    

For more detailed instructions on how to get started with BigQuery, check out the BigQuery quickstart guide.

**Data schema¶**

Linehaul writes an entry in a bigquery-public-data.pypi.file_downloads table for each download. The table contains information about what file was downloaded and how it was downloaded. Some useful columns from the table schema include:

  

Column

Description

Examples

timestamp

Date and time

2020-03-09 00:33:03 UTC

file.project

Project name

pipenv, nose

file.version

Package version

0.1.6, 1.4.2

details.installer.name

Installer

pip, bandersnatch

details.python

Python version

2.7.12, 3.6.4

**Useful queries¶**

Run queries in the BigQuery web UI by clicking the “Compose query” button.

Note that the rows are stored in a partitioned table, which helps limit the cost of queries. These example queries analyze downloads from recent history by filtering on the timestamp column.

**Counting package downloads¶**

The following query counts the total number of downloads for the project “pytest”.

#standardSQL
SELECT COUNT(\*) AS num\_downloads
FROM \`bigquery-public-data.pypi.file\_downloads\`
WHERE file.project = 'pytest'
  -- Only query the last 30 days of history
  AND DATE(timestamp)
    BETWEEN DATE\_SUB(CURRENT\_DATE(), INTERVAL 30 DAY)
    AND CURRENT\_DATE()

num\_downloads

26190085

To count downloads from pip only, filter on the details.installer.name column.

#standardSQL
SELECT COUNT(\*) AS num\_downloads
FROM \`bigquery-public-data.pypi.file\_downloads\`
WHERE file.project = 'pytest'
  AND details.installer.name = 'pip'
  -- Only query the last 30 days of history
  AND DATE(timestamp)
    BETWEEN DATE\_SUB(CURRENT\_DATE(), INTERVAL 30 DAY)
    AND CURRENT\_DATE()

num\_downloads

24334215

**Package downloads over time¶**

To group by monthly downloads, use the TIMESTAMP_TRUNC function. Also filtering by this column reduces corresponding costs.

#standardSQL
SELECT
  COUNT(\*) AS num\_downloads,
  DATE\_TRUNC(DATE(timestamp), MONTH) AS \`month\`
FROM \`bigquery-public-data.pypi.file\_downloads\`
WHERE
  file.project = 'pytest'
  -- Only query the last 6 months of history
  AND DATE(timestamp)
    BETWEEN DATE\_TRUNC(DATE\_SUB(CURRENT\_DATE(), INTERVAL 6 MONTH), MONTH)
    AND CURRENT\_DATE()
GROUP BY \`month\`
ORDER BY \`month\` DESC

 

num\_downloads

month

1956741

2018-01-01

2344692

2017-12-01

1730398

2017-11-01

2047310

2017-10-01

1744443

2017-09-01

1916952

2017-08-01

**Python versions over time¶**

Extract the Python version from the details.python column. Warning: This query processes over 500 GB of data.

#standardSQL
SELECT
  REGEXP\_EXTRACT(details.python, r"\[0-9\]+\\.\[0-9\]+") AS python\_version,
  COUNT(\*) AS num\_downloads,
FROM \`bigquery-public-data.pypi.file\_downloads\`
WHERE
  -- Only query the last 6 months of history
  DATE(timestamp)
    BETWEEN DATE\_TRUNC(DATE\_SUB(CURRENT\_DATE(), INTERVAL 6 MONTH), MONTH)
    AND CURRENT\_DATE()
GROUP BY \`python\_version\`
ORDER BY \`num\_downloads\` DESC

 

python

num\_downloads

3.7

18051328726

3.6

9635067203

3.8

7781904681

2.7

6381252241

null

2026630299

3.5

1894153540

**Getting absolute links to artifacts¶**

It’s sometimes helpful to be able to get the absolute links to download artifacts from PyPI based on their hashes, e.g. if a particular project or release has been deleted from PyPI. The metadata table includes the path column, which includes the hash and artifact filename.

Note

The URL generated here is not guaranteed to be stable, but currently aligns with the URL where PyPI artifacts are hosted.

SELECT
  CONCAT('https://files.pythonhosted.org/packages', path) as url
FROM
  \`bigquery-public-data.pypi.distribution\_metadata\`
WHERE
  filename LIKE 'sampleproject%'

url

https://files.pythonhosted.org/packages/eb/45/79be82bdeafcecb9dca474cad4003e32ef8e4a0dec6abbd4145ccb02abe1/sampleproject-1.2.0.tar.gz

https://files.pythonhosted.org/packages/56/0a/178e8bbb585ec5b13af42dae48b1d7425d6575b3ff9b02e5ec475e38e1d6/sampleproject\_nomura-1.2.0-py2.py3-none-any.whl

https://files.pythonhosted.org/packages/63/88/3200eeaf22571f18d2c41e288862502e33365ccbdc12b892db23f51f8e70/sampleproject\_nomura-1.2.0.tar.gz

https://files.pythonhosted.org/packages/21/e9/2743311822e71c0756394b6c5ab15cb64ca66c78c6c6a5cd872c9ed33154/sampleproject\_doubleyoung18-1.3.0-py2.py3-none-any.whl

https://files.pythonhosted.org/packages/6f/5b/2f3fe94e1c02816fe23c7ceee5292fb186912929e1972eee7fb729fa27af/sampleproject-1.3.1.tar.gz

**Caveats¶**

In addition to the caveats listed in the background above, Linehaul suffered from a bug which caused it to significantly under-report download statistics prior to July 26, 2018. Downloads before this date are proportionally accurate (e.g. the percentage of Python 2 vs. Python 3 downloads) but total numbers are lower than actual by an order of magnitude.

**Additional tools¶**

Besides using the BigQuery console, there are some additional tools which may be useful when analyzing download statistics.

**google-cloud-bigquery¶**

You can also access the public PyPI download statistics dataset programmatically via the BigQuery API and the google-cloud-bigquery project, the official Python client library for BigQuery.

from google.cloud import bigquery

\# Note: depending on where this code is being run, you may require
\# additional authentication. See:
\# https://cloud.google.com/bigquery/docs/authentication/
client \= bigquery.Client()

query\_job \= client.query("""
SELECT COUNT(\*) AS num\_downloads
FROM \`bigquery-public-data.pypi.file\_downloads\`
WHERE file.project = 'pytest'
  -- Only query the last 30 days of history
  AND DATE(timestamp)
    BETWEEN DATE\_SUB(CURRENT\_DATE(), INTERVAL 30 DAY)
    AND CURRENT\_DATE()""")

results \= query\_job.result()  \# Waits for job to complete.
for row in results:
    print("{} downloads".format(row.num\_downloads))

**pypinfo¶**

pypinfo is a command-line tool which provides access to the dataset and can generate several useful queries. For example, you can query the total number of download for a package with the command pypinfo package_name.

Install pypinfo using pip.

python3 \-m pip install pypinfo

Usage:

$ pypinfo requests
Served from cache: False
Data processed: 6.87 GiB
Data billed: 6.87 GiB
Estimated cost: $0.04

| download\_count |
| -------------- |
|      9,316,415 |

**pandas-gbq¶**

The pandas-gbq project allows for accessing query results via Pandas.

**References¶**

1

PyPI Download Counts deprecation email

2

PyPI BigQuery dataset announcement email

CVE-2022-25024: Analyzing PyPI package downloads — Python Packaging User Guide

A heap overflow bug exists FreeImage before 1.18.0 via ofLoad function in PluginJPEG.cpp.

*   Summary
*   Files
*   Reviews
*   Support
*   Mailing Lists
*   Code
*   Tickets ▾
    *   Feature Requests
    *   Patches
    *   Bugs
    *   Support Requests
*   News
*   Discussion
*   FreeImage

Menu ▾ ▴

Status: open

Owner: nobody

Labels: None

Priority: 5

Updated: 2021-08-26

Created: 2021-08-26

Private: No

> tl;dr:  
> When Load SOF(Start Of Frame) JPEG files, "components" of SOF is read from file. FreeImage alloc memory size is controlled by "components" , but sometime FreeImage replace default size to alloc. if "components" is greater than default in alloc function, it will writing data more than the allocated memory.

When the program reads a JPEG file, it will be handed to the Load function of the 'PluginJPEG.cpp' file,  
in Load function, we alloc memory by FreeImage\_AllocateHeader(), and write memory in jpeg\_read\_scanlines(). And the "components" is read from file.

static FIBITMAP \* DLL\_CALLCONV
Load(FreeImageIO \*io, fi\_handle handle, int page, int flags, void \*data) {
    if (handle) {
......
        struct jpeg\_decompress\_struct cinfo;
......
            // step 3: read handle parameters with jpeg\_read\_header()

            jpeg\_read\_header(&cinfo, TRUE);                //<---- read components from file
......
            if((cinfo.output\_components \== 4) && (cinfo.out\_color\_space \== JCS\_CMYK)) {
......
            } else {
                // RGB or greyscale image              v------------ alloc memory function
                dib \= FreeImage\_AllocateHeader(header\_only, cinfo.output\_width, cinfo.output\_height, 8 \* cinfo.output\_components, FI\_RGBA\_RED\_MASK, FI\_RGBA\_GREEN\_MASK, FI\_RGBA\_BLUE\_MASK);
......
            }
......
            if((cinfo.out\_color\_space \== JCS\_CMYK) && ((flags & JPEG\_CMYK) != JPEG\_CMYK)) {
......
            } else if((cinfo.out\_color\_space \== JCS\_CMYK) && ((flags & JPEG\_CMYK) \== JPEG\_CMYK)) {
......
            } else {
                // normal case (RGB or greyscale image)

                while (cinfo.output\_scanline < cinfo.output\_height) {
                    JSAMPROW dst \= FreeImage\_GetScanLine(dib, cinfo.output\_height \- cinfo.output\_scanline \- 1);

                    jpeg\_read\_scanlines(&cinfo, &dst, 1);       // <---------- write memory
                }

                // step 7b: swap red and blue components (see LibJPEG/jmorecfg.h: #define RGB\_RED, ...)
                // The default behavior of the JPEG library is kept "as is" because LibTIFF uses 
                // LibJPEG "as is".
......
    }

    return NULL;
}

FreeImage\_AllocateHeader() alloc memory by "output\_components", but FreeImage\_AllocateHeader function maybe replace default size with it.

FIBITMAP \* DLL\_CALLCONV
FreeImage\_AllocateHeader(BOOL header\_only, int width, int height, int bpp, unsigned red\_mask, unsigned green\_mask, unsigned blue\_mask) {
    return FreeImage\_AllocateBitmap(header\_only, NULL, 0, FIT\_BITMAP, width, height, bpp, red\_mask, green\_mask, blue\_mask);
}

static FIBITMAP \* 
FreeImage\_AllocateBitmap(BOOL header\_only, BYTE \*ext\_bits, unsigned ext\_pitch, FREE\_IMAGE\_TYPE type, int width, int height, int bpp, unsigned red\_mask, unsigned green\_mask, unsigned blue\_mask) {
.......
    // check pixel bit depth
    switch(type) {
        case FIT\_BITMAP:
            switch(bpp) {
                case 1:
                case 4:
                case 8:
                    break;
                case 16:
                    need\_masks \= TRUE;
                    break;
                case 24:
                case 32:
                    break;
                default:
                    bpp \= 8;                                                                    //<---- change bpp there
                    break;
            }
            break;
.......
        size\_t dib\_size \= FreeImage\_GetInternalImageSize(header\_only || ext\_bits, width, height, bpp, need\_masks);          //<\---- calc alloc size by bpp(num\_components)

        if(dib\_size \== 0) {
            // memory allocation will fail (probably a malloc overflow)
            free(bitmap);
            return NULL;
        }

        bitmap-\>data \= (BYTE \*)FreeImage\_Aligned\_Malloc(dib\_size \* sizeof(BYTE), FIBITMAP\_ALIGNMENT);                       //<\---- real alloc function
.......
}

in FreeImage\_GetInternalImageSize(), it calc alloc size

static size\_t 
FreeImage\_GetInternalImageSize(BOOL header\_only, unsigned width, unsigned height, unsigned bpp, BOOL need\_masks) {
    size\_t dib\_size \= sizeof(FREEIMAGEHEADER);
......
    if(!header\_only) {
        const size\_t header\_size \= dib\_size;

        // pixels are aligned on a 16 bytes boundary
        dib\_size += (size\_t)CalculatePitch(CalculateLine(width, bpp)) \* (size\_t)height;    //<---alloc size by line lenght \* height, line lenght calc by width and bpp(num\_components)
......
    }

    return dib\_size;
}

but when we use jpeg\_read\_scanlines, the read size calc by "num\_components", usually equal "output\_components".

METHODDEF(void)
null\_convert (j\_decompress\_ptr cinfo,
          JSAMPIMAGE input\_buf, JDIMENSION input\_row,
          JSAMPARRAY output\_buf, int num\_rows)
{
  register JSAMPROW outptr;
  register JSAMPROW inptr;
  register JDIMENSION count;
  register int num\_comps \= cinfo\->num\_components;
  JDIMENSION num\_cols \= cinfo\->output\_width;
  int ci;

  while (\--num\_rows \>= 0) {
    /\* It seems fastest to make a separate pass for each component. \*/
    for (ci \= 0; ci < num\_comps; ci++) {
      inptr \= input\_buf\[ci\]\[input\_row\];
      outptr \= output\_buf\[0\] + ci;
      for (count \= num\_cols; count \> 0; count\--) {
    \*outptr \= \*inptr++; /\* don't need GETJSAMPLE() here \*/        //<---- copy size calc by width(output\_width) and bpp(num\_components)
    outptr += num\_comps;
      }
    }
    input\_row++;
    output\_buf++;
  }
}

callstack

FreeImage.dll!null\_convert(jpeg\_decompress\_struct \* cinfo, unsigned char \* \* \* input\_buf, unsigned int input\_row, unsigned char \* \* output\_buf, int num\_rows)
FreeImage.dll!sep\_upsample(jpeg\_decompress\_struct \* cinfo, unsigned char \* \* \* input\_buf, unsigned int \* in\_row\_group\_ctr, unsigned int in\_row\_groups\_avail, unsigned char \* \* output\_buf, unsigned int \* out\_row\_ctr, unsigned int out\_rows\_avail)
FreeImage.dll!process\_data\_simple\_main(jpeg\_decompress\_struct \* cinfo, unsigned char \* \* output\_buf, unsigned int \* out\_row\_ctr, unsigned int out\_rows\_avail)
FreeImage.dll!jpeg\_read\_scanlines(jpeg\_decompress\_struct \* cinfo, unsigned char \* \* scanlines, unsigned int max\_lines)
FreeImage.dll!Load(FreeImageIO \* io, void \* handle, int page, int flags, void \* data)
FreeImage.dll!FreeImage\_LoadFromHandle(FREE\_IMAGE\_FORMAT fif, FreeImageIO \* io, void \* handle, int flags)
FreeImage.dll!FreeImage\_Load(FREE\_IMAGE\_FORMAT fif, const char \* filename, int flags)

Finally, it will cause the heap overflow if we make the "components" not in \[1, 2, 3\]. Moreover, it may has the risk of arbitrary code execution.

windbg:

ModLoad: 10000000 105ee000   D:\\temp\\\_zzz\\FreeImage.dll
ModLoad: 752b0000 752c4000   C:\\Windows\\SysWOW64\\VCRUNTIME140.dll
ModLoad: 755e0000 75643000   C:\\Windows\\SysWOW64\\WS2\_32.dll
ModLoad: 77050000 7710f000   C:\\Windows\\SysWOW64\\RPCRT4.dll
(5cb0.349c): Break instruction exception \- code 80000003 (first chance)
eax\=00000000 ebx\=00000000 ecx\=608e0000 edx\=00000000 esi\=77442044 edi\=7744260c
eip\=774e1b72 esp\=0133f5c8 ebp\=0133f5f4 iopl\=0         nv up ei pl zr na pe nc
cs\=0023  ss\=002b  ds\=002b  es\=002b  fs\=0053  gs\=002b             efl\=00000246
ntdll!LdrpDoDebuggerBreak+0x2b:
774e1b72 cc              int     3
0:000\> g
(5cb0.349c): Access violation \- code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
\*\*\* WARNING: Unable to verify checksum for D:\\temp\\\_zzz\\FreeImage.dll
eax\=08539007 ebx\=00000009 ecx\=00000880 edx\=073590c8 esi\=00000721 edi\=00000000
eip\=1009aec5 esp\=0133f6cc ebp\=00000000 iopl\=0         nv up ei pl nz na pe nc
cs\=0023  ss\=002b  ds\=002b  es\=002b  fs\=0053  gs\=002b             efl\=00010206
FreeImage!jinit\_color\_deconverter+0x935:
1009aec5 8808            mov     byte ptr \[eax\],cl          ds:002b:08539007\=??
0:000\> db eax\-0x20
08538fe7  00 00 00 00 00 80 00 00-00 c0 c0 c0 c0 c0 80 c0  ................
08538ff7  c0 c0 c0 c0 d0 d0 d0 80\-d0 ?? ?? ?? ?? ?? ?? ??  .........???????
08539007  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
08539017  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
08539027  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
08539037  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
08539047  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
08539057  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
0:000\> kv
 \# ChildEBP RetAddr      Args to Child              
00 0133f6d8 1009a383     00000808 073536cc 00000000 FreeImage!jinit\_color\_deconverter+0x935 (FPO: \[Uses EBP\] \[5,0,1\])
01 0133f704 100991d3     073536c0 07354fe0 07355008 FreeImage!jinit\_upsampler+0x263 (FPO: \[Uses EBP\] \[7,1,4\])
02 0133f734 10085bfa     0133f830 0133fa18 0133f754 FreeImage!jinit\_d\_main\_controller+0x1f3 (FPO: \[Uses EBP\] \[4,0,4\])
03 0133f74c 1002283f     00000000 0133fa18 00000001 FreeImage!jpeg\_read\_scanlines+0x8a (FPO: \[3,0,1\])
04 0133fa48 1000d9de     0133fa9c 07339fc8 ffffffff FreeImage!jpeg\_freeimage\_dst+0x1b1f (FPO: \[5,183,3\])
05 0133fa7c 1000da60     00000002 0133fa9c 07339fc8 FreeImage!FreeImage\_LoadFromHandle+0x8e (FPO: \[Uses EBP\] \[4,3,2\])
06 0133faa8 00b0107b     00000002 07335fee 00000000 FreeImage!FreeImage\_Load+0x50 (FPO: \[3,4,2\])

TestFile:

data:image/jpeg;base64,/9j/2wADAP/KACMICAgICAkBETLvESEDESH/QSF5IgAxETExFDExIjExETEx/9oACAF5AAAAAA\==

**2 Attachments**

**Discussion**

Log in to post a comment.

CVE-2021-40265: FreeImage / Bugs / #337 A heap_overflow on PluginJPEG.cpp when Load() SOF(Start Of Frame) JPEG

Cross Site Scripting (XSS) vulnerability in Cacti 1.2.21 via crafted POST request to graphs_new.php.

XSS vulnerability in Cacti https://github.com/Cacti/cacti version v1.2.21

The path of the vulnerability. In file https://github.com/Cacti/cacti/blob/develop/graphs\_new.php

//line 40
switch (get\_request\_var('action')) {
              case 'save':
                            form\_save();
 
//line 117 the source in $\_POST
function form\_save() {
              if (isset\_request\_var('save\_component\_graph')) {
                            /\* summarize the 'create graph from host template/snmp index' stuff into an array \*/
                            foreach ($\_POST as $var => $val) {
                            //…..
                           if (strpos($var, 'sgg\_') !== false) {
                                                         // Note: the source in $snmp\_query\_id then function store\_get\_selected\_dq\_index will be called
                                                         $snmp\_query\_id = str\_replace('sgg\_', '', $var);
                                                        store\_get\_selected\_dq\_index($snmp\_query\_id);
                                          }
                            }
            //…….
}
// line 100
Note the source in $snmp\_query\_id
function store\_get\_selected\_dq\_index($snmp\_query\_id) {
              // ….
              } elseif (isset\_request\_var('sgg\_' . $snmp\_query\_id)) {
                             // Note: get\_filter\_request\_var will be called
                            $selected = get\_filter\_request\_var('sgg\_' . $snmp\_query\_id);
              }
//….
}

In file https://github.com/Cacti/cacti/blob/develop/lib/html\_utility.php

//line 424
// the source in the argument $name
function get\_filter\_request\_var($name, $filter = FILTER\_VALIDATE\_INT, $options = array()) {
//….
//line 503
if ($value === false) {
                                          if ($filter == FILTER\_VALIDATE\_IS\_REGEX) {
                                                         //….
                                          } else {
                                                         // Note: function die\_html\_input\_error will be called
                                                         die\_html\_input\_error($name, get\_nfilter\_request\_var($name));
                                          }
                            }

In file https://github.com/Cacti/cacti/blob/develop/lib/html\_validate.php

//line 47
// Note: the source in $variable
function die\_html\_input\_error($variable = '', $value = '', $message = '') {
              //….
 
              if ($message == '') {
              // Note: the $message will include the $variable as I will explain later, then it will be printed
              $message = \_\_('Validation error for variable %s with a value of %s.  See backtrace below for more details.', $variable, $value);
              }
              //Note: the print of the $message
             $variable = ($variable != '' ? ', Variable:' . $variable : '');
              $value    = ($value    != '' ? ', Value:'    . $value    : '');
 
              if (defined('CACTI\_CLI\_ONLY')) {
                            cacti\_debug\_backtrace('Validation Error' . $variable . $value, false);
                            print $message . PHP\_EOL;
                            exit(1);
              } elseif (isset\_request\_var('json')) {
                            cacti\_debug\_backtrace('Validation Error' . $variable . $value, false);
                            print json\_encode(
                                          array(
                                                         'status' => '500',
                                                         'statusText' => \_\_('Validation Error'),
                                                         'responseText' => $message
                                          )
                            );
              } else {
                            cacti\_debug\_backtrace('Validation Error' . $variable . $value, true);
 
                            print "<table style='width:100%;text-align:center;'><tr><td>$message</td></tr></table>";
                            bottom\_footer();
              }
 
              exit;
}
}

In file https://github.com/Cacti/cacti/blob/develop/include/global\_languages.php

//line 432
function \_\_() {
              global $l10n;
 
              $args = func\_get\_args();
              $num  = func\_num\_args();
 
              //….
                            else{
                                          $args\[0\] = \_\_gettext($args\[0\]);
                            }
 
                            /\* process return string against input arguments \*/
                            return \_\_uf(call\_user\_func\_array('sprintf', $args));
              }
}
 
//line 393
function \_\_gettext($text, $domain = 'cacti') {
              //….
 
              if (!isset($translated)) {
                            $translated = $text;
              } 
 
              //…..
              return \_\_uf($translated);
}
 
//line 428
function \_\_uf($text) {
              return str\_replace('%%', '%', $text);
}

The vulnerability is confirmed by the developers. The email sent on 18/06/2022.

Tag

#js