Friday, February 17, 2017

Altium Unified Components - Catch 'Em All!

Until recently, the Altium Designer "official" component libraries (known as "Unified Components") were only available to Altium subscribers - they are now however available to anyone by filling out a short survey (no registration or subscription required). After doing this, you receive an email with a download link. This process gets old and tedious very quickly every time you need a new component library... Yes, that's right: Altium does not offer a single download for all the libraries! So I decided to take matters into my own hands and wrote a Python script that uses some XPath and Selenium magic to automatically download all 578 libraries in less than 5 minutes :) This script actually relies on the fact that the library download links are contained in the page source! Thanks Altium!

Requirements:
Usage (Windows):
1. Download and install Python.
2. Install Selenium:
pip install selenium
3. Download the PhantomJS pre-built executable and place in the same directory as script.
4. Run Python script:
python download_all_altium_unified_components.py

I am of course providing the script source for educational purposes only... :)
Any suggestions or feedback on this script are welcome too.

download_all_altium_unified_components.py

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import re
import concurrent.futures
import urllib.request
import urllib.parse
import posixpath
import os

BASE_URL                = "https://designcontent.live.altium.com/"
START_URL               = BASE_URL + "UnifiedComponents/"
PAGE_LOAD_WAIT_S        = 20
DOWNLOAD_TIMEOUT_S      = 60
MAX_DOWNLOAD_WORKERS    = 8

def wait_for_page_to_load(driver, timeout_s):
    WebDriverWait(driver, timeout_s).until(EC.presence_of_element_located((By.ID, "xUCS_frmItemsView")))

def get_next_page_url(driver):
    try:
        link = driver.find_element_by_xpath("//div[@id='xUCS_frmItemsView']//div[@id='UCS_frmItemsView_TextLabel2']/div/div[1]/following-sibling::a[1][@class='HijaxLink']")
        return BASE_URL + link.get_attribute("href").replace(START_URL, "")
    except:
        return ""

def get_content_urls(driver):
    divs = driver.find_elements_by_xpath("//div[contains(@id, 'UCS_frmItemsView_Fields') and @id!='UCS_frmItemsView_FieldsPopup']")
    return [re.search('ZipURL`(.*\.zip)`[\S+\n\r\s]+', div.get_attribute("title")).group(1) for div in divs]

def download(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()

# Start WebDriver (choose one!)
print("Starting WebDriver...")
#driver = webdriver.Firefox()
driver = webdriver.PhantomJS(service_args=['--load-images=no'])
print("WebDriver started.")

# Open page
driver.get(START_URL)
# Wait for page to load
wait_for_page_to_load(driver, PAGE_LOAD_WAIT_S)
# Get URLs
urls_all = list()
urls = get_content_urls(driver)
urls_all += urls
print("%r: ZIP download links retrieved from page: %d (Total: %d)" % (driver.current_url, len(urls), len(urls_all)))

# Do remaining pages
next_page_url = get_next_page_url(driver)
while next_page_url != "":
    # Open page
    driver.get(next_page_url)
    # Wait for page to load
    wait_for_page_to_load(driver, PAGE_LOAD_WAIT_S)
    # Get URLs
    urls = get_content_urls(driver);
    urls_all += urls
    print("%r: ZIP download links retrieved from page: %d (Total: %d)" % (driver.current_url, len(urls), len(urls_all)))
    # Get next page URL (if any)
    next_page_url = get_next_page_url(driver)

# Quit WebDriver
driver.quit()

# Write URLs to file
urls_file = open('altium_unified_components_urls.txt', 'w')
for url in urls_all:
    urls_file.write("%s\n" % url)
urls_file.close()

# Download all URLs
error_count = 0
print("%d files will be downloaded..." % len(urls_all))
with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_DOWNLOAD_WORKERS) as executor:
    # Start the download operations and mark each future with its URL
    future_to_url = {executor.submit(download, url, DOWNLOAD_TIMEOUT_S): url for url in urls_all}
    for future in concurrent.futures.as_completed(future_to_url):
        url = future_to_url[future]
        try:
            data = future.result()
            print('%r file is %d bytes' % (url, len(data)))
            # Get filename from URL
            path = urllib.parse.urlsplit(url).path
            filename = urllib.parse.unquote(posixpath.basename(path))
            # Save file
            savepath = "download/" + filename
            os.makedirs(os.path.dirname(savepath), exist_ok=True)
            with open(savepath, "wb") as file:
                file.write(data)
        except Exception as e:
            print('%r generated an exception: %s' % (url, e))
            error_count += 1

# Done!
if error_count == 0:
    print("Finished successfully.")
else:
    print("Finished with %d error(s)." % error_count)

Update 1 (June 25 2017):
Download link for Unified Components archive as obtained from the above script. Enjoy!

Update 2 (December 9 2017):
14 new libraries have since been added by Altium bringing the total to 592 (updated library archive download link). New libraries:
TE_Connectivity_ D-Shaped_Connectors.zip
TE_Connectivity_Audio,_Video_and_High_Speed_Serial_Connectors.zip
TE_Connectivity_Automotive_Connectors_(Car,_Truck,_Bus,_and_Off-Road).zip
TE_Connectivity_Card_Edge_Connectors_and_Sockets.zip
TE_Connectivity_Humidity_Sensors.zip
TE_Connectivity_Modular_Jacks_and_Plugs_(RJ45,_RJ11,_RJ25).zip
TE_Connectivity_Passive_Components.zip
TE_Connectivity_PCB_Connectors.zip
TE_Connectivity_PCB_Terminals.zip
TE_Connectivity_Power_Connectors,_Contacts,_and_Terminals.zip
TE_Connectivity_Pressure_Sensors.zip
TE_Connectivity_Relays,_Contactors_and_Switches.zip
TE_Connectivity_RF_and_Coax_Connectors.zip
TE_Connectivity_Terminal_Blocks_and_Strips.zip  


Update 3 (January 13 2018):
I have slightly modified the script to download all available online content: Unified Components, NanoBoard Examples and Reference Designs (Note: Template Designs seem to only be available via Altium Vault).
download_altium_design_content.py
altium_design_content_20180113.zip [888 libraries, 1.34 GB]

3 comments:

  1. Hello,
    this is awesome!
    I would like to know if there are any further updates on this project.
    I tried this procedure (with latest packages of Python and phantomJS), and it seems it doesn't work anymore.
    Thank You very much

    ReplyDelete
  2. Great Job,
    Really usefull!
    Thank You very much

    ReplyDelete