Recent Entries 10
- pattern minor 112d agoChecking HTTP headers with asyncio and aiohttpThis is one of my first attempts to do something practical with `asyncio`. The task is simple: Given a list of URLs, determine if the content type is HTML for every URL. I've used `aiohttp`, initializing a single "session", ignoring SSL errors and issuing HEAD requests to avoid downloading the whole endpoint body. Then, I simply check if `text/html` is inside the `Content-Type` header string: ``` import asyncio import aiohttp @asyncio.coroutine def is_html(session, url): response = yield from session.head(url, compress=True) print(url, "text/html" in response.headers["Content-Type"]) if __name__ == '__main__': links = ["https://httpbin.org/html", "https://httpbin.org/image/png", "https://httpbin.org/image/svg", "https://httpbin.org/image"] loop = asyncio.get_event_loop() conn = aiohttp.TCPConnector(verify_ssl=False) with aiohttp.ClientSession(connector=conn, loop=loop) as session: f = asyncio.wait([is_html(session, link) for link in links]) loop.run_until_complete(f) ``` The code works, it prints (the output order is inconsistent, of course): ``` https://httpbin.org/image/svg False https://httpbin.org/image False https://httpbin.org/image/png False https://httpbin.org/html True ``` But, I'm not sure if I'm using `asyncio` loop, wait and coroutines, `aiohttp`'s connection and session objects appropriately. What would you recommend to improve?
- pattern minor 112d agoUsing Task.Run(..) in a .NET propertySummary I have a normal class which has a normal property but that property has access to an async backing field. I'm not sure if this is the best/acceptable way to access the asyc method for this backing field. Details I've got a C# library that helps testing the .NET `HttpClient` class. This class has a property `public HttpContent HttpContent` which is access two private backing fields which some code uses later on. One of these private backing fields is populated via an `async` method. This, being a property, isn't `async`. When the value of this property is `set`, I extract the actual string content from it by calling the method `_httpContent.ReadAsStringAsync()` and also remember a reference to this instance via a class-global variable. The actual code in question is this ``` private string _httpContentSerialized; .... _httpContentSerialized = _httpContent == null ? "*" : Task.Run(_httpContent.ReadAsStringAsync).Result; ``` As such, I'm not sure if the code I have (which seems to work, mind you) is the best way to do this and I'm hoping someone can review this (as opposed to asking on SO for a 'how to do this' answer). Extra Info to consider: Being a library I have no idea how/where my library will be consumed/used: - A console app? - An ASP.NET app? With this in mind, I'm not sure if the code I have written is acceptable and won't cause deadlocks for a .NET application. Other suggestions have been to use `_httpContent.ReadAsStringAsync().ConfigureAwait(false).GetAwaiter().GetResult();` Here's some more of the code: ``` public class HttpMessageOption { private HttpContent _httpContent; private string _httpContentSerialized; public HttpContent HttpContent { get { return _httpContent; } set { _httpContent = value; _httpContentSerialized = _httpContent == null ? "*" : Task.Run(_httpContent.ReadAsStringAsync).Result; } } }
- pattern minor 112d agoNode.js module using Promises (client for KeePassHttp)This is my first Node module, as well as the first time using Promises in Javascript. It is a client for the KeePass plugin "KeePassHTTP" to expose passwords securely, which I am planning on using to pass credentials to gulp. All feedback welcome! ``` // Code based on https://github.com/belaviyo/keepass-macpass-helper // Keepass HTTP protocol documentation - https://github.com/pfn/keepasshttp const sjcl = require('./sjcl').sjcl; // not using npm module because it doesn't have support for cbc const nconf = require('nconf'); const rp = require('request-promise-native'); let port = null; let key = null; let id = null; function iv(len = 16) { let iv = []; for (let i = 0; i { let request = { RequestType: 'test-associate', TriggerUnlock: false, }; request = verify(request); post(request) .then(response => { if (response && response['Success']) { resolve(response); } else { reject(response); } }) .catch(response => reject(response)); }); } function associate() { return new Promise((resolve, reject) => { let request = { RequestType: 'associate', Key: key }; request = verify(request); post(request) .then(response => { if (response && response['Success']) { id = response['Id']; nconf.set('id', id); nconf.save(); resolve(response); } else { reject(response); } }) .catch(response => reject(response)); }); } function logins(url) { return new Promise((resolve, reject) => { let request = { RequestType: 'get-logins', TriggerUnlock: 'false', SortSelection: 'false', }; request = verify(request); const iv = request['Nonce']; request['Url'] = encrypt(url, iv); post(request) .then(response => { if (response && response['Entries']) {
- pattern minor 112d agoMultithreaded search for solutions to an inequalityThis question is related to my previous question on the brute-force search for a solution to an unsolved mathematical inequality: $$3^{k}-2^{k}\left\lfloor \left({\tfrac {3}{2}}\right)^{k}\right\rfloor >2^{k}-\left\lfloor \left({\tfrac {3}{2}}\right)^{k}\right\rfloor -2$$ The following code, tested on Ubuntu 16.04, compiles with no warnings or errors using the flags `-Wall -std=c++14 -pthread -Ofast` using the g++ compiler. I'm using `-Ofast` because I am trying to squeeze every ounce of performance out of this code (more info here). I am not 100% sure if this is the proper, fastest, implementation of multithreading. Since I'm new to multithreading, I used this SO question, this page, and this page for learning how to multithread. For questions about other parts of the code, I explain it way more on my previous question (like why I have a completely separate `if`-`else` check). Let me know how I can improve my use of multithreading in this program. ``` #include //#include (already included from cpp_dec_float.hpp) #include #include typedef boost::multiprecision::number> arbFloat; enum returnID {success = 0, precisionExceeded = 1}; arbFloat calcThreeK(const arbFloat & k){ return pow(3, k); } arbFloat calcTwoK(const arbFloat & k){ return pow(2, k); } int main(){ arbFloat k, threeK, twoK; bool isSolution = false; for(k = 6; !isSolution; ++k){ std::future futureThreeK = std::async(std::launch::async, calcThreeK, k); std::future futureTwoK = std::async(std::launch::async, calcTwoK, k); threeK = futureThreeK.get(); twoK = futureTwoK.get(); isSolution = threeK - twoK * floor(threeK / twoK) > twoK - floor(threeK / twoK) - 2; } if(pow(3, k) - (pow(2, k) * floor(pow(1.5, k))) <= pow(2, k) - floor(pow(1.5, k)) - 2){ std::cout << "Solution at k = " << k << ".\n"; return returnID::success; } else { std::cout << "Error: Precision exceeded at k = " << k << ".\n";
- pattern minor 112d agoLightweight asynchronous event library in C - Threadpool moduleI have finished writing a C library whose purpose is to provide a simple API for asynchronously executing functions, waiting for events on file descriptors and waiting for timeouts. The whole library is published here https://github.com/ernacktob/asyncio. The entire library is probably too large to be reviewed in a single question, so I am trying to split things up into the separate modules. In this question I am interested in feedback for the threadpool module. The purpose of this module is to implement an API for submitting functions to be executed by a fixed pool of worker threads, or optionally, spawn a separate thread just for this function. The overall structure of this module consists of a global queue of tasks to be executed by a fixed set of worker threads, and another global queue for "contractor" threads (which are threads spawned for an individual task). The user calls threadpool_dispatch to submit a function to the thread pool, and a structure containing all the relevant information is stored in the appropriate queue. The worker threads wait on the queue, and the first to wake will take the first task from the queue and execute it. The purpose of contractor threads is to handle cases where the task might take a long time to complete, and would stall the worker queue. So this provides the user the option to spawn a dedicated thread for that task, bypassing the worker queue. The module uses the underlying POSIX pthread library for actually creating/cancelling threads, as well as mutex and condition variables. The API provides a threadpool_handle_t, which is an opaque pointer to a threadpool_handle struct. This handle is used for the user to do things such as blocking until the task is completed (threadpool_join) or cancelling the task (threadpool_cancel). The public interface to this module is the following (taken from the include/threadpool.h header). The public functions are all defined at the bottom of the source file, with the rest being internal
- pattern minor 112d agoTCP retarder for network delay simulationI would like to simulate delay over network, for that I have implemented a simple retarder (=introduce random delay while maintaining order of messages) using Boost to handle sending itself. It appears to work as intended. However, since parallel programming is not trivial especially when coupled with networking, I would like to hear feedback more experienced C++ programmer. Mostly, I am concerned about the parallelization(deadlocks, resource leaks) and possible side effects on the communication I might have missed. ``` class SendRetarder { typedef std::pair args_t;//to whom and what std::queue message_queue;//messages to be sent std::mutex m; std::condition_variable cv; const size_t max_wait = 10000; void send_loop() { while (true) { boost::system::error_code ec; args_t args; size_t wait; { std::unique_lock lk(m); cv.wait(lk, [&] { return !message_queue.empty(); }); wait = (rand() % max_wait) / message_queue.size();//to prevent unchecked growing args = message_queue.front(); message_queue.pop(); } std::this_thread::sleep_for(std::chrono::milliseconds(wait)); boost::asio::write(args.first->socket_, boost::asio::buffer(&args.second, sizeof(message_t)), ec); if (ec) std::cerr lk(m); message_queue.push(args_t(client, msg)); } cv.notify_one(); } }; ``` where `message_t` is the contents of the message consisting of integer types and `session_ptr` is a smart pointer wrapper around socket shared with client itself: ``` struct message_t { msg_id_t message_id; tag_t message_tag; sender_t sender_id; union { uint32_t value; bool success; } data; } typedef std::shared_ptr session_ptr; class Session : public std::enable_shared_from_this { tcp::socket socket_; sender_t name_; //... } ```
- pattern minor 112d agoC++ Wrapper for cURL: Multithreading and serializing asynchronous opsI did this library to help me from one of my projects and decided to publish it on GitHub hoping people might find it useful, convenient and easy to use in their projects too. It's a header-only library. The library design is inspired by some popular Java code conventions, like naming of classes and the popular usage of chaining call of methods. As of now, there are only two HTTP methods I made from `CppUrl` class, which are the `GET` and `POST` with the two corresponding "fetch" functions to execute it either on main thread, or on another thread namely: `execute()` (synchronous) and `async()` (asynchronous). I haven't really tested this on some complicated scenarios so if I may ask, what could probably go wrong here? I could smell some data corruption or some deadlocks when this has been used on an intensive and complicated task, but it might just me. I want the user of this library to type as less as possible of code. So, what do you think of using `std::map` as a function parameter for passing files on a `POST` method? Is it suitable (considering the context)? If not, can you suggest another constructive and simpler way? ``` class CppUrl { using ResponseCallback = std::function; public: class MissingHttpMethodException : std::exception { public: std::string what() { return "No HTTP method used"; } }; CppUrl() : task(nullptr) { if (curlInstanceCount == 0) { curl_global_init(CURL_GLOBAL_ALL); } handle = curl_easy_init(); if (handle) { curlInstanceCount++; } } CppUrl(const CppUrl&) = delete; CppUrl& operator= (CppUrl const&); ~CppUrl() { curlInstanceCount--; curl_easy_cleanup(handle); cleanTask(); handle = nullptr; // busy = false; }); return *this; } /** * Perform a request with a POST method * @param const std::string &url: URL to be requested * @param s
- pattern moderate 112d agoAsynchronous TCP serverAfter some investigation, I implemented an asynchronous TCP server as per the following example. During my investigation I was unable to find an example that cleanly shuts down the server; after some experimenting I was able to furnish my code with this functionality. I would appreciate a review of the service code for whether I might be doing something stupid/dangerous. (Please note that I have stripped out parameter validation, etc. in order to improve readability). ``` public class AsyncTcpServer : IDisposable { public class DataReceivedEventArgs : EventArgs { public NetworkStream Stream { get; private set; } public DataReceivedEventArgs(NetworkStream stream) { Stream = stream; } } public event EventHandler OnDataReceived; public AsyncTcpServer(IPAddress address, int port) { _listener = new TcpListener(address, port); } public void Start() { _listener.Start(); _isListening = true; WaitForClientConnection(); } public void Stop() { _isListening = false; _listener.Stop(); } public void Dispose() { Stop(); } private void WaitForClientConnection() { _listener.BeginAcceptTcpClient(HandleClientConnection, _listener); } private void HandleClientConnection(IAsyncResult result) { if (!_isListening) { return; } var server = result.AsyncState as TcpListener; var client = _listener.EndAcceptTcpClient(result); WaitForClientConnection(); OnDataReceived?.Invoke(this, new DataReceivedEventArgs(client.GetStream())); } private readonly TcpListener _listener; private volatile bool _isListening = false; } ``` The following test verifies the asynchronous nature of the service (test completes under 10 seconds for 5x client connections blocking for 5 seconds each). ``` [TestMethod] public void TestSendRec
- pattern minor 112d agoUsing SendAsync to send multiple emailsMy method for sending emails with `Send()` is getting very slow when I send more than ten messages. Sample 1 (1 sec) 2 (2 sec) 3 (4 sec) 4 (8 sec) and so on... so I started to try `SendAsync()` ``` public Boolean sendSMTPMail(string subject, string body, customer[] recipients, string attachmentFileName = null) { string css = "*{font-family: Arial, Helvetica, sans-serif;}"; SmtpClient client = new SmtpClient(); //some client settings.. MailMessage mm = new MailMessage(); mm.IsBodyHtml = true; mm.From = new MailAddress(MainForm.from, MainForm.fromName, Encoding.UTF8); mm.Subject = subject; foreach (var recipient in recipients) { Application.DoEvents(); mm.To.Clear(); mm.To.Add(new MailAddress(recipient.getEmail())); mm.AlternateViews.Add(getEmbeddedImages(css + doReplacements(body, recipient))); try { //client.Send(mm); client.SendAsync(mm, "userToken"); //client.SendMailAsync(mm); } catch (Exception) { return false; } } return true; } ``` Additional Information about my methods: `getEmbeddedImages` replaces `imagePath` with needed `cid:xxxx` and adds the image so it can be sent as a part of the email and not only as attachment. `doReplacements` replaces strings in my body to get the email text more recipient specific: ``` Hello my name is %name%, I am %age% years old. ``` will be translated to ``` Hello my name is John Doe, I am 100 years old. ``` Questions: - Is this the right way to do it? - Is it important to send different userTokens? - For what is this token needed anyway? Obviously it doesn't matter what I fill in there.
- pattern minor 112d agoHTTP downloader using BeastI have written a small HTTP downloader using - `boost::asio` - Beast library (proposed to be included in Boost) - `network::uri` library for handling URIs It's nowhere near completion but I would like to get some feedback from you guys. I wanted to use as much of async interface as I could. The idea is that it gets a `std::vector` with URLs to fetch and then it goes callback after callback to finally print the response data on screen. This review's purpose is that I would like to e.g. add support for HTTPS and while doing that I would like to write unit tests mocking somehow (and preferably splitting this class) network (asio) dependencies. Downloader.hpp ``` #include #include #include #include #include #include #include class Downloader { public: Downloader(const std::vector &urls); void go(); private: void read_handler(const boost::system::error_code &ec, std::shared_ptr> response, std::shared_ptr); void connect_handler(const boost::system::error_code &ec, const network::uri &uri); void queue_read(const boost::system::error_code &ec); void resolve_handler(const boost::system::error_code &ec, boost::asio::ip::tcp::resolver::iterator it, const network::uri &uri); std::vector uris; boost::asio::io_service ioservice; boost::asio::ip::tcp::socket tcp_socket; boost::asio::ip::tcp::resolver resolv; }; ``` Downloader.cpp ``` #include "Downloader.hpp" #include #include #include #include #include #include using namespace std::placeholders; // for _1, _2 etc. using namespace boost::asio; using namespace boost::asio::ip; Downloader::Downloader(const std::vector &urls) : tcp_socket(ioservice), resolv(ioservice) { std::transform(urls.begin(), urls.end(), std::back_inserter(uris), [this](const std::string &url) { std::error_code ec; network::uri u(url, ec); if(ec) { ioservice.post([=] { std::cout > response