I'm just now moving my main workflows off openai over to local models and I'm starting to find that these smaller models main failure mode is that they will accept edgecases with the goal of being helpful.
Especially in extraction tasks. This appears as inventing data or rationalizing around clear roadblocks.
My biggest hack so far is giving them an out named "edge_case" and telling them it is REALLY helpful if they identify edgecases. Simply renaming "fail_closed" or "dead_end" options to "edge_case" with helpful wording causes qwen models to adhere to their prompting more.
It feels like there are 100s of these small hacks that people have to have discovered... why isn't there a centralized place where people are recording these learnings?
Exactly. Using lmstudio you can get structured output like with openai... I have everything in a zod schema then wrote an adapter to both openai and lmstudio instance.
I run lmstudio for ease of use on several mac studios that are fronted by a small token aware router that estimates resource usage on the mac studios.
Lots of optimization left there, but the systems are pinned most of the time so not focused on that at the moment as the gpus are the issue not the queuing.
I would like to hear more about your set up if you’re willing. Is the token aware router you’re using publicly available or something you’ve written yourself?
It isn't open... but drop me an email and I can send you it. Basically just tracks a list of known lmstudios on the network, queries their models every 15 seconds and routes to the ones who have the requested models loaded in a FIFO queue tracking the number of tokens/model (my servers are uniform... m4 max 128gb studios but could also track the server) and routes to the one that has just finished. I used to have it queue one just as it was expected to finish but was facing timeout issues due to an edgecase.
I have one that's currently still cooking, I have good experimental validation for it but I need to go back and tune the latency and improve the install story. It should help any model quite a bit but you have to hack other agents to integrate it into their api call machinery, I have a custom agent I've built that makes it easy to inject though.
ADK is a nice framework but it's still stuck in the agent as atomic chatbot that goes out and does stuff model. The reality is you want your agents to all be running within an orchestrator service because it's much more efficient in pretty much every way. The way you separate concerns here is to have the agent emit intents, and have those intents go to a queue to be acted upon safely and securely by executors. This system is more secure, performant and observable than the ADK setup.
DuckDB is mind blowingly awesome. It is like SQLite, lightweight, embeddable, serverless, in-memory database, but it's optimized to be columnar (analytics optimized). It can work with files that are in filesystem, S3 etc without copying (it just looks at the necessary regions in the file) by just doing `select * from 's3://....something.parquet'`. It support automatic compression and automatic indexing. It can read json lines, parquet, CSV, its own db format, sqlite db, excel, Google Sheets... It has a very convenient SQL dialect with many QoL improvements and extensions (and is PostgreSQL compatible). Best of all: it's incredibly fast. Sometimes it's so fast that I find myself puzzled "how can it possibly analyze 10M rows in 0.1 seconds?" and I find it difficult to replicate the performance in pure Rust. It is an extremely useful tool. In the last year, it has become one of my use-everyday tools because the scope of problems you can just throw DuckDB at is gigantic. If you have a whole bunch of structured data that you want to learn something about, chances are DuckDB is the ideal tool.
PS: Not associated with DuckDB team at all, I just love DuckDB so much that I shill for them when I see them in HN.
I'm sorry, I must be exceptionally stupid (or haven't seriously worked in this particular problem domain and thus lacking awareness), but I still can't figure out the use cases from this feature list.
What sort of thing should I be working on, to think "oh, maybe I want this DuckDB thing here to do this for me?"
I guess I don't really get the "that you want to learn something about" bit.
I’m not the person you asked, but here are some random, assorted examples of “structured data you want to learn something about”:
- data you’ve pulled from an API, such as stock history or weather data,
- banking records you want to analyze for patterns, trends, unauthorized transactions, etc
- your personal fitness data, such as workouts, distance, pace, etc
- your personal sleep patterns (data retrieved from a sleep tracking device),
- data you’ve pulled from an enterprise database at work — could be financial data, transactions, inventory, transit times, or anything else stored there that you might need to pull and analyze.
Here’s a personal example: I recently downloaded a publicly available dataset that came in the form of a 30 MB csv file. But instead of using commas to separate fields, it used the pipe character (‘|’). I used DuckDB to quickly read the data from the file. I could have actually queried the file directly using DuckDB SQL, but in my case I saved it to a local DuckDB database and queried it from there.
My dumb guy heuristic for DuckDB vs SQLite is something like:
- Am I doing data analysis?
- Is it read-heavy, write-light, using complex queries over large datasets?
- Is the dataset large (several GB to terabytes or more)?
- Do I want to use parquet/csv/json data without transformation steps?
- Do I need to distribute the workload across multiple cores?
If any of those are a yes, I might want DuckDB
- Do I need to write data frequently?
- Are ACID transactions important?
- Do I need concurrent writers?
- Are my data sets tiny?
- Are my queries super simple?
If most of the first questions are no and some of these are yes, SQLite is the right call
On way to think about it is SQLite for columnar / analytical data.
It works great against local files, but my favorite DuckDB feature is that it can run queries against remote Parquet files, fetching just the ranges of bytes it needs to answer the query using HTTP range queries.
This means you can run eg a count(*) against a 15GB parquet file from your laptop and only fetch a few hundred KBs of data (if that).
Small intro, It's a relational database for analytical data primarily. It's an "in-process" database meaning you can import certain files at runtime and query them. That's how it differs primarily from regular relational systems.
for the average developer I think the killer feature is allowing you to query over whatever data you want (csv, json, parquet, even gsheets) as equals, directly from their file form - can even join across them
It has great CSV and JSON processing so I find it's almost better thought of as an Excel-like tool vs. a database. Great for running quick analysis and exploratory work. Example: I need to do some reporting mock-ups on Jira data; DuckDB sucks it all in (or queries exports in place), makes it easy to clean, filter, pivot, etc. export to CSV
If you're developing in the data space you should consider your "small data" scenarios (ex: the vast majority of our clients have < 1GB of analytical data; Snowflake, etc. is overkill). Building a DW that exists entirely in a local browser session is possible now; that's a big deal.
As an American living in Europe, it is fun to see the filter bubble Europeans live in with regards to how the US is framed in the media here and even in common coversation. I was talking to my French buddy about it and we were comparing our filter bubbles. Please realize that there is a filter bubble that all of us live in.
I consume a lot of American social media as a European, what about his comment was wrong?
Fox News, Twitter, Facebook etc have really done a number on the US (and the rest of the world to some extent). The lack of regulation of these companies have brought us to the Post-Truth world we're living in now.
I'm not a Trump fan but the immigration situation was getting a bit out of hand. It was a bit do you want the US to be the US you knew or have the whole of south and central America move in.
Sure, The Dark Lord is evil, but did you notice how the
Mayor of Michel Delving failed to do anything about that rabbit plague that decimated the Longbottom tobacco harvest two years ago?
The Mayor of Michel Delving flat out refused to help the tobacco growers of Longbottom, letting the rabbits have their way. The Dark Lord will surely treat them better! Or at least kind of ignore them. Or, well, he does seem supportive of the rabid rabbit leader, but just as often as not he's just incoherent on the topic of the plight of the Longbottom tobacco growers, and I'll take that as a good sign!
Do not know who that guy is, just googled and found out he was Biden's buddy hence irrelevant and it does not change my opinion of they supported genocide enablers.
Leaving policies aside, on purely moral standards, it seems hard to "both sides" this.
The people that attempt this were incredibly bad faith i.e. for example equating Hillary's "[Trump] knows he’s an illegitimate president" which is calling out shady voter suppression tactics and the fact that Trump did not win the popular vote with the staunch denial of the 2020 election result by Trump to this day and the organisation of a (failed) plot to remain in power.
These situations aren't really comparable. Clinton questioned procedural issues and the popular vote outcome, while Trump's actions following the 2020 election (including his continued denial of results and attempts to overturn them) represent an unprecedented challenge to American democratic institutions. It's deeply concerning how many people continue to accept claims that have been thoroughly debunked by election officials, courts, and independent observers. This erosion of trust in democratic processes and willingness to embrace demonstrably false narratives suggests a troubling shift in American political culture. It's a fallen nation.
Can you give a brief comparison of the different views? I’m European and watch/read a lot of US news sources from both sides and still think “European” about Trump.
You’re missing the political viewpoint of 45% of the country if you’re not main-lining right-wing radio and Fox News. No, stop—don’t google what they just said, imagine you just believe it and see where that takes you.
[edit] if you want root-cause for how we got here, look into media ownership laws in the US, and into the Citizens United court case (check out the 5-4 podcast for a fun/horrifying take). Single entities can own unlimited reach of media, which didn’t used to be the case, as recently as the very early 2000s IIRC, and Citizens opened up unlimited corporate spending in elections, with exactly the implications you’d expect for e.g. foreign spending on US elections. Er, I mean, the actual root cause is kinda the system of elections the slave states pushed into the constitution, if you wanna go way back, but the proximate cause of the current political landscape is that.
Note: Running out the door, here is a brief summary that hasn't been deeply considered.
======
Example: I casually commented at the gym that I thought Trump would win about a week ago and went into a very long conversation/debate with several people in the gym.
I asked each of them to show me their filter bubbles... once I had explained it to them. 2 Spaniards, 2 Andorran, 1 French guy. The general consensus seemed to be that Trump was a major step back socially and would pollute the environment, not care about climate change, etc.
Their information came from traditional news, Instagram/Tiktok and one from Reddit/HN/X. The only one that was even open to hearing my views during the conversation was the guy who read Reddit/HN/X. Everyone else had their minds made up.
Today at the gym, I had the 2nd part of this conversation with 3 of the guys that were there this morning. The general take away that the guy who read Reddit/HN/X summed up is one of: "Europe has yet to reach "peak tolerance" where it appears the US is already there." This realization was came to due to the issues of immigrants in both Spain/France who don't assimilate which was a hot button issue for them.
I kinda agree with this sentiment. I think that Europe (at least the part I'm in) doesn't seem to have a great immune system for people who abuse the system and generally punishes tall poppies both socially and economically. The US is completely different, tall poppies are celebrated and if you fail or get sick you have no safety net.
For me the difference in world views is best summed up by what people are focusing on. Climate, equality, and tolerance are the key issues I see pushed heavily in Europe... I see this as stemming from the "tall poppy" syndrome that is prevalent in both Spain/France. In the US, people care about other things and are generally focused on things that directly impact them. That is what this election was about. Less focus on perceived injustices or injustices of the past and more focus on making the future better with something different.
Is Europe a great place? For sure... but it has wildly different problems and world views than the US does. I think it is hard to appreciate that until you've bene immersed in both cultures long enough.
In general, I'd just say there is more nuance to everything than our brains can handle. My little mission has been to try and bring back nuance into the conversation. Black and white thinking is lazy. Nuance exists, find it and challenge your filter bubble.
I'm excited to see how this casual gym conversation continues.
I'm Italian and the views I have here (left/center left/center friends/relatives) is that he's a crazy nut job that's gonna hurt europe as well as limiting abortion rights and similar things. I'd say there's not much talk about economy because we don't understand shit in our own economy either, let alone a foreign one... Except that we think the rich are gonna get much richer and he's gonna cut social spending (ie, less Healthcare and similar things) and he's gonna crackdown on immigration. Not expressing our own views on it, that's just what i think it's talked about Trump here
Oh, he's also a misogynistic, convicted felon that spews lies. This is partially our view in my bubble, if I had to say it entirely I would get flagged lol
