ruạṛ
a O��f�x � @ sB d dl Z d dlZd dlZd dlZd dlZddlmZ d dlm Z d dl m Z d dl m Z ddlmZ dd lmZ dd lmZ d d lmZ d dlmZmZ d dlmZmZ d dlmZm Z d dl m Z d dl m Z d dl!m"Z" d dl#m$Z$ d dl%m&Z& dZ'd Z(dZ)g d�Z*d�+e&�Z,e�� Z-d�+e,e-�Z.e�/e0�Z1G dd � d �Z2dS ) � N� )�md5)�Pool)�stat)� InternalError� )�DEFAULT_PART_SIZE)�MEBIBYTE)�BufferedPartReader� )�models� )� ServiceError�MultipartUploadError)�RequestException�ConnectTimeout)�Timeout�ConnectionError)�Queue)� Semaphore)�six)� is_fips_mode)� __version__i � )�opc_sse_customer_algorithm�opc_sse_customer_key�opc_sse_customer_key_sha256zOracle-PythonSDK/{}z�Client Version: {}, OS Version: {}, See https://docs.oracle.com/iaas/Content/API/Concepts/sdk_troubleshooting.htm for common issues and steps to resolve them. If you need to contact support, or file a GitHub issue, please include this full error message.c @ s� e Zd Zdd� Zedd� �Zedd� �Zdd� Zd d � Zd d d�Z dd� Z dd� Z dd� Z dd� Z dd� Zdd� Zdd � Zd d � Zd S ) �MultipartObjectAssemblerc K s� || _ t| _d|v r |d | _d| _d|v r6|d | _d| _d|v rN|d | _d| _d|v rf|d | _d| _d|v r~|d | _d| _d|v r�|d | _d| _d|v r�|d | _d| _ d |v r�|d | _ d| _ d |v r�|d | _ d| _ d |v r�|d | _ t | _ d |v �r&|d d k�r td��|d | _ d|v �rD|d du �rDd| _ t| _d|||g d�| _d|v �rn|d nd| _i | _tD ] }||v �r~|| | j|< �q~dS )u� MultipartObjectAssembler provides a simplified interaction when uploading large objects using multi-part uploads. An assembler can be used to begin a new upload, or resume a previous one. PLEASE NOTE that for oci versions < 2.23.2 and >= 2.18.0, the timeout for the object storage client is overwritten to `None` for all operations which call object storage. For this reason, the operations are NOT thread-safe, and you should provide the MultipartObjectAssembler class with its own Object Storage client that isn't used elsewhere. For more information please see `Known Issues <https://docs.oracle.com/en-us/iaas/tools/python/latest/known-issues.html>`_ :param ObjectStorageClient object_storage_client: A configured Object Storage client. :param str namespace_name: The namespace containing the bucket in which to store the object. :param str bucket_name: The name of the bucket in which to store the object. :param str object_name: The name to use for the object in Object Storage, or existing object name if resuming. :param int part_size (optional): Override the default part size of 128 MiB, value is in bytes. :param str if_match (optional): The entity tag of the object to match. :param str if_none_match (optional): The entity tag of the object to avoid matching. The only valid value is ‘*’, which indicates that the request should fail if the object already exists. :param str content_type (optional): The content type of the object to upload. :param str content_language (optional): The content language of the object to upload. :param str content_encoding (optional): The content encoding of the object to upload. :param str content_disposition (optional): The content disposition of the object to upload. :param str cache_control (optional): The cache control for the object to upload. :param dict metadata (optional): A dictionary of string to string values to associate with the object to upload :param bool allow_parallel_uploads (optional): Whether or not this MultipartObjectAssembler supports uploading parts in parallel. Defaults to True. :param int parallel_process_count (optional): The number of worker processes to use in parallel for performing a multipart upload. Default is 3. :param str opc_sse_customer_algorithm (optional): The encryption algorithm to use with the customer provided encryption key. :param str opc_sse_customer_key (optional): The base64-encoded 256-bit encryption key to use to encrypt the objects uploaded by this MultipartObjectAssembler. :param str opc_sse_customer_key_sha256 (optional): The base64-encoded SHA256 hash of the encryption key. :param str opc_sse_kms_key_id (optional): The OCID of a master encryption key used to call the Key Management Service to generate a data encryption key or to encrypt or decrypt a data encryption key. :param str storage_tier: (optional) The storage tier that the object should be stored in. If not specified, the object will be stored in the same storage tier as the bucket. Allowed values are: "Standard", "InfrequentAccess", "Archive" � part_sizeN�if_match� if_none_match� content_type�content_language�content_encoding�content_disposition� cache_control�metadata� storage_tier�parallel_process_countr z,parallel_process_count must be at least one.�allow_parallel_uploadsFr )�uploadId� namespace� bucketName� objectName�parts�opc_sse_kms_key_id)�object_storage_clientr r r r r! r" r# r$ r% r& r' � DEFAULT_PARALLEL_PROCESS_COUNTr( � ValueError�DEFAULT_MAX_RETRIES� max_retries�manifestr/ � ssec_params�SSEC_PARAM_NAMES)�selfr0 �namespace_name� bucket_name� object_name�kwargs� param_name� r> ��/sparta/input/_build_configuration/image_build+validate/lib/bmcenv/lib64/python3.9/site-packages/oci/object_storage/transfer/internal/multipart_object_assembler.py�__init__, sd S � z!MultipartObjectAssembler.__init__c C s� t � rXz t�� }W q` tyT } z*t�d�|�� tjddd�}W Y d}~q`d}~0 0 nt�� }|}t j | dd��<}t |||�}|� t �}|dkr�q�|�|� q�W d � n1 s�0 Y t�|�� ��d �S ) a4 Calculate the base64 encoded MD5 hash for a part of a file. :param str file_path: Path to the file :param int offset: Offset where the part starts in the file :param int chunk: Number of bytes in the part :return: Base64 encoded MD5 hash :rtype: str �bAn exception occur due to {}. Fallback to using hashlib.new('md5', usedforsecurity=false) for md5.r F��usedforsecurityN�rb��mode� �utf-8)r �MD5r r �logger�warning�format�hashlib�new�io�openr �read�READ_BUFFER_SIZE�update�base64� b64encode�digest�decode) � file_path�offset�chunkr �ex�m�f�bpr�partr> r> r? � calculate_md5� s & *z&MultipartObjectAssembler.calculate_md5c C s� d}t | t�rd}npt | t�r$d}n`t | t�r4d}nPt | t�rDd}n@t | t�r�| jdks�| jdks�| jdkrv| jdks�| jdkr�d}|S )az Determines if the service should attempt to retry an operation based on the type of exception retry if timeout Connection error unknown client exception: status == -1 server exception: status >= 500 Potential edge case: status == 409 :param e: Exception :return: Boolean FTi� ���i� �ConcurrentObjectUpdatei� )� isinstancer r r r r �status�code)�e� retryabler> r> r? �_is_exception_retryable� s 2z0MultipartObjectAssembler._is_exception_retryablec K s~ t j|d dd��X}t||d |d �}|j|d |d |d |d |d |fi |d ��W d � S 1 sp0 Y d S ) N�part_file_pathrD rE rY �sizer+ r: r; � upload_id�part_num� new_kwargs)rO rP r � upload_part)r8 r0 r< � file_objectr^ r> r> r? �_upload_part_call� s ��z*MultipartObjectAssembler._upload_part_callc C s� t j|dd���}|�dt j� |�� }|�dt j� d}|�� |k r�|| }| j|||| jkrd| jn|d� || j7 }|�|t j� q8W d � n1 s�0 Y dS )z� Splits a file into parts and adds all parts to an internal list of parts to upload. The parts will not be uploaded to the server until upload is called. :param string file_path: Path of file to upload in parts rD rE r )rY rj N)rO rP �seek�SEEK_END�tell�SEEK_SET�add_part_from_filer )r8 rX ro �endrY � remainingr> r> r? �add_parts_from_file s z,MultipartObjectAssembler.add_parts_from_filer Nc C s0 |du rt |�j}| jd �||||d�� dS )aM Add a part to internal list of parts to upload. The part will not be uploaded to the server until upload is called. :param str file_path: Path to file :param offset: Offset of part in file :param size: Size of the part in bytes :param part_hash: Base64 encoded MD5 hash of the part Nr. )rX rY rj �hash)r �st_sizer5 �append)r8 rX rY rj � part_hashr> r> r? ru s �z+MultipartObjectAssembler.add_part_from_filec K sh d|v r |d | j d< |�d� | j d s2td��| jj| j d | j d | j d | j d fi |�� dS )a& Abort the multipart upload :param str upload_id (optional): The upload id to abort. If not provided, will attempt to abort the upload created internally by new_upload. :param str opc_client_request_id: (optional) The client request ID for tracing rk r* z"Cannot abort without an upload id.r+ r, r- N)r5 �popr2 r0 �abort_multipart_upload)r8 r< r> r> r? �abort+ s ��z MultipartObjectAssembler.abortc K sL d|v r |d | j d< |�d� | j d du r6td��i }d|v rX|d |d< |�d� d}|�r8| jj| j d | j d | j d | j d fi |��}| j d }|jD ]�}|jd }d | k r�t|�k �rn n@|| }|d |jkr�td� |jt ���|j |d< |j |d< q�|t|�kr�td��q�|j }|j|d< q\| jf i |�� dS )a� Resume uploading a multipart object to Object Storage :param str upload_id (optional): The upload id to resume. If not provided, will attempt to resume the upload created internally by new_upload. :param function progress_callback (optional): Callback function to receive the number of bytes uploaded since the last call to the callback function. :param str opc_client_request_id: (optional) The client request ID for tracing rk r* Nz#Cannot resume without an upload id.�progress_callbackTr+ r, r- r. r ra rj z]Cannot resume upload with different part size. Parts were uploaded with a part size of {} MiB�etag�opc_md5zTThere are more parts on the server than parts to resume, please check the upload ID.�page)r5 r} r2 r0 �list_multipart_upload_parts�data� part_number�lenrj rL r r� r � has_next_page� next_page�upload) r8 r< � upload_kwargsr� �responser. r_ � part_index� manifest_partr> r> r? �resumeA s@ �� z MultipartObjectAssembler.resumec K sb | j d rtd��t�� }| j d |_| jr4| j|_| jrB| j|_| jrP| j|_| jr^| j|_| j rl| j |_ | j r�i }t � | j �D ](\}}|� d�s�||d| < q�|||< q�|| _ | j |_ | jr�| j|_d}d|v r�|d }i }|r�||d< | jr�| j|d< | j�r| j|d< | j�r | j|d < |�| j� | jj| j d | j d |fi |��}|jj| j d< dS ) z� Create a new multipart upload in Object Storage and add the upload id to the manifest :param str opc_client_request_id: (optional) The client request ID for tracing. r* zECannot call new_upload again once an upload has already been created.r- z opc-meta-N�opc_client_request_idr r r/ r+ r, )r5 � RuntimeErrorr � CreateMultipartUploadDetails�objectr! r" r# r$ r% r& r � iteritems� startswithr' r r r/ rS r6 r0 �create_multipart_uploadr� rk )r8 r< �request�processed_metadata�key�value�client_request_idr� r> r> r? � new_uploadu sX ��z#MultipartObjectAssembler.new_uploadc K s� d|d i}d|v r |d |d< d|v r4|d |d< |� | j� |d du rj| �|d |d |d �|d< d}d |v r~|d }d |v�r�|r�|j| j| j| jd | jd | jd | jd |d |d |d ||d� }n�| j}|dk�r~zH| j| j| jd | jd | jd | jd |d |d |d ||d� }W nH t�yv } z.| � |��r`|dk�r`|d8 }n� W Y d}~q�d}~0 0 �q~q�|j dk�r�|j d |d< t |j d �|d < d|v �r�|d |d � dS )aV Upload a part to Object Storage :param int part_num: Sequence number for where this part belongs in the multipart object. :param dict part: Part dictionary containing the following keys: "file_path" (str), "hash" (str), "offset" (int), and "size" (int) :param str opc_client_request_id: (optional) The client request ID for tracing. :param function progress_callback (optional): Callback function to receive the number of bytes uploaded since the last call to the callback function. � content_md5ry r� r/ NrX rY rj �retry_strategyr� r+ r, r- r* ) r+ r: r; rk ri rY rj rl rm r r �� r� �opc-content-md5r� ) rS r6 r` �make_retrying_callrp r0 r5 r4 � Exceptionrh rd �headers�str) r8 rl r_ r< rm r� r� �remaining_triesrf r> r> r? � _upload_part� sd � � z%MultipartObjectAssembler._upload_partc K sB �z"�z�t � r`z t�� }W qh ty\ } z*t�d�|�� tjddd�}W Y d }~qhd }~0 0 nt�� }|� |� dt � |� � �� d�i}d|v r�|d |d< | j}|dk�rjzD| jj| jd | jd | jd | jd |d t�|�fi |��}W np t�yb } zV| �| ��r$|d k�r$|d 8 }n*d|v �rLd|d d< |d d �| � � W Y d } ~ q�d } ~ 0 0 �qjq�|jdk�r�| jd �|jd t|jd �|d�� d|v �r�|d t|�� W nP t�y } z6d|v �r�d|d d< |d d �| � � W Y d } ~ n d } ~ 0 0 W d|v �r>|d � � nd|v �r<|d � � 0 d S )NrA r FrB r� rH r� r r+ r, r- r* r � shared_dict�should_continue� exceptionsr� r. r� r� )r� r� rl r� � semaphore) r rI r r rJ rK rL rM rN rS rT rU rV rW r4 r0 rn r5 rO �BytesIOr� rh �putrd r{ r� r� r� �release) r8 rl � part_bytesr< r\ r[ rm r� r� rf r> r> r? �_upload_stream_part sl ��& �� � � z,MultipartObjectAssembler._upload_stream_partc sZ �j d du rtdt ��t�jd�}|�� �fdd�t�j d �� |�� |�� dS )aO Upload an object to Object Storage :param function progress_callback (optional): Callback function to receive the number of bytes uploaded since the last call to the callback function. :param str opc_client_request_id: (optional) The client request ID for tracing. r* N�CCannot call upload before initializing an upload using new_upload. �� processesc s" �j f | d d | d d�� ��S )Nr r )rl r_ )r� )� part_tuple�r< r8 r> r? �<lambda>Q rG z1MultipartObjectAssembler.upload.<locals>.<lambda>r. ) r5 r� �$UPLOAD_MANAGER_DEBUG_INFORMATION_LOGr r( �map� enumerate�close�join)r8 r< �poolr> r� r? r� B s �z MultipartObjectAssembler.uploadc K s | j d d u rtdt ��t| jd�}t| j�}dt� d�}d}|�� }||d< ||d< d}|r�tj r�t |d �r�|j � | j �} n |� | j �} |�� t| �dkr�|�| j|| f|� |d 7 }t| �| j ko�|d }q\|�� |�� |d �� s�t|d d ��| j d jdd� d� d S )Nr* r� r� T)r� r� r r� r� �bufferr r� r� )�error_causes_queuer. c S s | d S )Nrl r> )r_ r> r> r? r� � rG z8MultipartObjectAssembler.upload_stream.<locals>.<lambda>)r� )r5 r� r� r r( r r �copyr �PY3�hasattrr� rQ r �acquirer� � apply_asyncr� r� r� �emptyr �sort) r8 � stream_refr< r� r� r� � part_counter�apply_async_kwargs� keep_reading� read_bytesr> r> r? � upload_streamV s0 z&MultipartObjectAssembler.upload_streamc K s� | j d du rtd��t�� }g }g }t| j d �D ]F\}}t�� }|d |_d|v rl|d |_|�|� q4|�|d � q4||_ ||_ | j j | j d | j d | j d | j d |fi |��}|S ) z~ Commit the multipart upload. :return: A Response object with data of type None :rtype: None r* NzeCannot call commit before initializing an upload using new_upload or resuming an upload using resume.r. r r� r+ r, r- ) r5 r� r � CommitMultipartUploadDetailsr� � CommitMultipartUploadPartDetailsrl r� r{ �parts_to_commit�parts_to_excluder0 �commit_multipart_upload) r8 r< �commit_detailsr� r� �partNumr_ �detailr� r> r> r? �commit� s. ��z MultipartObjectAssembler.commit)r NN)�__name__� __module__� __qualname__r@ � staticmethodr` rh rp rx ru r r� r� r� r� r� r� r� r> r> r> r? r + s( � 4<Q@7r )3rO rM rT �logging�platform� r rI �multiprocessing.dummyr �osr �cryptography.exceptionsr � constantsr r �buffered_part_readerr r r� r r �oci.exceptionsr r � oci._vendor.requests.exceptionsr r �oci._vendor.six.moves.queuer � threadingr � oci._vendorr �oci.fipsr �versionr rR r1 r3 r7 rL �CLIENT_VERSION� OS_VERSIONr� � getLoggerr� rJ r r> r> r> r? �<module> s:
cải xoăn